justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 20917.41@dunegpschedd01.fnal.gov

Jobsub ID20917.41@dunegpschedd01.fnal.gov
Workflow ID404
Stage ID1
User namemsiden@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes8387559424 (7999 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-08-05 03:22:43
SiteCZ_FZU
EntryNova_CZ_FZU_osgce1
Last heartbeat2025-08-05 08:45:24
From worker nodeHostnamemilan25.farm.particle.cz
cpuinfoAMD EPYC 7543 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit172800 (48 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-08-05 04:32:59
Input fileshd-protodune:np04hd_raw_run027087_0120_dataflow1_datawriter_0_20240614T154943.hdf5
JobscriptExit code0
Real time4h (15114s)
CPU time4h (14632s = 96%)
Max RSS bytes3634388992 (3466 MiB)
Outputting started2025-08-05 08:44:54
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/msiden/channel_map_test/fnal/00404/1/001/pdhd_processing_1_20917_41_1754368387_stage1_stage2_tfile.root
https://fndcadoor.fnal.gov:2880/dune/scratch/users/msiden/channel_map_test/fnal/00404/1/001/pdhd_processing_1_20917_41_1754368387_stage1_stage2.root
Finished2025-08-05 08:45:24
Saved logsjustin-logs:20917.41-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

Saver: q=2.46687e+06 n=1096470 tag=wiener
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=5859 keeping as is
Retagger: tagging trace set: wiener with 62358 traces, 62358 summary
Retagger: tagging trace set: gauss with 56744 traces, 0 summary
wclsFrameSaver saving cooked to 5859 ticks
wclsFrameSaver: saving 56744 traces tagged "gauss"
FrameSaver: q=1.2948e+06 n=764267 tag=gauss
wclsFrameSaver: saving 62358 traces tagged "wiener"
FrameSaver: q=1.40413e+06 n=805429 tag=wiener
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=5859 keeping as is
Retagger: tagging trace set: wiener with 68033 traces, 68033 summary
Retagger: tagging trace set: gauss with 60741 traces, 0 summary
wclsFrameSaver saving cooked to 5859 ticks
wclsFrameSaver: saving 60741 traces tagged "gauss"
FrameSaver: q=1.35929e+06 n=790619 tag=gauss
wclsFrameSaver: saving 68033 traces tagged "wiener"
FrameSaver: q=1.47389e+06 n=835942 tag=wiener
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=5859 keeping as is
Retagger: tagging trace set: wiener with 72198 traces, 72198 summary
Retagger: tagging trace set: gauss with 63347 traces, 0 summary
wclsFrameSaver saving cooked to 5859 ticks
wclsFrameSaver: saving 63347 traces tagged "gauss"
FrameSaver: q=1.5284e+06 n=764235 tag=gauss
wclsFrameSaver: saving 72198 traces tagged "wiener"
FrameSaver: q=1.74063e+06 n=816681 tag=wiener
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=5859 keeping as is
Retagger: tagging trace set: wiener with 88698 traces, 88698 summary
Retagger: tagging trace set: gauss with 76850 traces, 0 summary
wclsFrameSaver saving cooked to 5859 ticks
wclsFrameSaver: saving 76850 traces tagged "gauss"
FrameSaver: q=2.30172e+06 n=1077153 tag=gauss
wclsFrameSaver: saving 88698 traces tagged "wiener"
FrameSaver: q=2.34406e+06 n=1094024 tag=wiener
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=5859 keeping as is
Retagger: tagging trace set: wiener with 77240 traces, 77240 summary
Retagger: tagging trace set: gauss with 66764 traces, 0 summary
wclsFrameSaver saving cooked to 5859 ticks
wclsFrameSaver: saving 66764 traces tagged "gauss"
FrameSaver: q=1.96433e+06 n=879786 tag=gauss
wclsFrameSaver: saving 77240 traces tagged "wiener"
FrameSaver: q=2.0918e+06 n=900937 tag=wiener
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=5859 keeping as is
Retagger: tagging trace set: wiener with 64926 traces, 64926 summary
Retagger: tagging trace set: gauss with 57542 traces, 0 summary
wclsFrameSaver saving cooked to 5859 ticks
wclsFrameSaver: saving 57542 traces tagged "gauss"
FrameSaver: q=1.43649e+06 n=772486 tag=gauss
wclsFrameSaver: saving 64926 traces tagged "wiener"
FrameSaver: q=1.49593e+06 n=792250 tag=wiener
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=5859 keeping as is
Retagger: tagging trace set: wiener with 66544 traces, 66544 summary
Retagger: tagging trace set: gauss with 59979 traces, 0 summary
wclsFrameSaver saving cooked to 5859 ticks
wclsFrameSaver: saving 59979 traces tagged "gauss"
FrameSaver: q=1.33541e+06 n=798627 tag=gauss
wclsFrameSaver: saving 66544 traces tagged "wiener"
FrameSaver: q=1.41569e+06 n=830803 tag=wiener
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=5859 keeping as is
Retagger: tagging trace set: wiener with 66563 traces, 66563 summary
Retagger: tagging trace set: gauss with 58999 traces, 0 summary
wclsFrameSaver saving cooked to 5859 ticks
wclsFrameSaver: saving 58999 traces tagged "gauss"
FrameSaver: q=1.4465e+06 n=809218 tag=gauss
wclsFrameSaver: saving 66563 traces tagged "wiener"
FrameSaver: q=1.50527e+06 n=835381 tag=wiener
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=5859 keeping as is
Retagger: tagging trace set: wiener with 116665 traces, 116665 summary
Retagger: tagging trace set: gauss with 96528 traces, 0 summary
wclsFrameSaver saving cooked to 5859 ticks
wclsFrameSaver: saving 96528 traces tagged "gauss"
FrameSaver: q=3.21179e+06 n=1399627 tag=gauss
wclsFrameSaver: saving 116665 traces tagged "wiener"
FrameSaver: q=3.42856e+06 n=1438872 tag=wiener
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=5859 keeping as is
Retagger: tagging trace set: wiener with 68599 traces, 68599 summary
Retagger: tagging trace set: gauss with 58718 traces, 0 summary
wclsFrameSaver saving cooked to 5859 ticks
wclsFrameSaver: saving 58718 traces tagged "gauss"
FrameSaver: q=1.41433e+06 n=762440 tag=gauss
wclsFrameSaver: saving 68599 traces tagged "wiener"
FrameSaver: q=1.57485e+06 n=793847 tag=wiener
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=5859 keeping as is
Retagger: tagging trace set: wiener with 60934 traces, 60934 summary
Retagger: tagging trace set: gauss with 51690 traces, 0 summary
wclsFrameSaver saving cooked to 5859 ticks
wclsFrameSaver: saving 51690 traces tagged "gauss"
FrameSaver: q=1.19121e+06 n=661342 tag=gauss
wclsFrameSaver: saving 60934 traces tagged "wiener"
FrameSaver: q=1.29746e+06 n=662691 tag=wiener
05-Aug-2025 10:44:51 CEST  Closed output file "pdhd_processing_1_20917_41_1754368387_stage1_stage2.root"
05-Aug-2025 10:44:51 CEST  Closed input file "pdhd_processing_1_20917_41_1754368387_stage1.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              161.615       353.149       705.49        328.373       120.971        40     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.000480711   0.00372145     0.0123176    0.00253328    0.00307359       40     
produce:fembfilter:PDHDFEMBFilter                      0.565057      0.696157       1.02878      0.671725      0.0995949       40     
produce:wclsdatahd:WireCellToolkit                      33.5749       40.2223       54.5363       37.9793       5.64928        40     
produce:gaushit:GausHitFinder                          0.376227      0.728461       1.69899      0.655522      0.264344        40     
produce:nhitsfilter:NumberOfHitsFilter                0.00022624    0.000556213   0.00218894    0.000476681   0.000369129      40     
produce:reco3d:SpacePointSolver                        0.147091      0.654737       2.72863      0.535902      0.465475        40     
produce:hitpdune:DisambigFromSpacePoints               0.151796      0.565963       3.06819      0.421564      0.475046        40     
produce:pandora:StandardPandora                         5.19699       15.4585       48.3569       13.1083       8.79174        40     
produce:pandoraWriter:StandardPandora                  0.0973471     0.197319      0.441853      0.174773      0.081431        40     
produce:pandoraTrack:LArPandoraTrackCreation           0.655765       2.86582       17.1613       2.28296       2.54872        40     
produce:pandoraShower:LArPandoraShowerCreation         0.606034       3.47567       28.264        2.08781       4.32958        40     
produce:pandoracalo:Calorimetry                        0.302237       1.03293       4.6738       0.870553       0.69011        40     
produce:pandoracalonosce:Calorimetry                   0.278853      0.970036       3.39124       0.79513      0.541131        40     
produce:pandorapid:Chi2ParticleID                     0.00142157    0.00256046    0.00633798    0.00226774    0.00105291       40     
produce:pandoraShowercalo:ShowerCalorimetry            0.483492       2.70682       20.0603       1.81996       3.08994        40     
produce:pandoraShowercalonosce:ShowerCalorimetry       0.483627       2.61648       17.833        1.72429       2.77485        40     
produce:pandoraGnocchiCalo:GnocchiCalorimetry          0.0541503     0.126067      0.346501      0.107932      0.0589308       40     
produce:pandoraGnocchiCalonosce:GnocchiCalorimetry     0.0205076     0.0478737      0.1217       0.0409174     0.0234149       40     
produce:emtrkmichelid:EmTrackMichelId                   17.3768       30.2483       63.2736       29.7848       9.06149        40     
[art]:TriggerResults:TriggerResultInserter             1.611e-05    6.51749e-05   0.00036017    5.29405e-05    5.923e-05       40     
end_path:out1:RootOutput                               6.01e-06     2.17723e-05    5.45e-05      1.895e-05    1.1433e-05       40     
end_path:michelremoving:michelremoving                 0.0176545     0.134397      0.544809      0.126484      0.105888        40     
end_path:michelremovinggnocchi:michelremoving          0.0166241      0.13308      0.766413      0.123478      0.129435        40     
end_path:calibanagnocchi:CalibAnaTree                   88.8586       248.699       465.849       222.882       98.3115        40     
end_path:out1:RootOutput(write)                        0.488804      0.877996       2.0875       0.845904      0.285889        40     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 6057.48 MB
  Peak resident set size usage (VmHWM): 3634.39 MB
  Details saved in: 'mem.db'
====================================================================================================

{'art_out': 'pdhd_processing_1_20917_41_1754368387_stage1_stage2.root', 'tfile_out': 'pdhd_processing_1_20917_41_1754368387_stage1_stage2_tfile.root', 'start_time': 1754369322.718993, 'end_time': 1754383493.4891803}
justIN time: 2025-08-05 23:17:28 UTC       justIN version: 01.04.00