justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 297921.3@dunegpschedd02.fnal.gov

Jobsub ID297921.3@dunegpschedd02.fnal.gov
Workflow ID12719
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:58:26
SiteIT_CNAF
EntryCMSHTPC_T1_IT_CNAF_condor_ce02
Last heartbeat2026-02-04 01:11:23
From worker nodeHostnamewn-s04-06-11-04.cr.cnaf.infn.it
cpuinfoAMD EPYC 7351 16-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 23:45:21
Input filesvd-protodune:np02vd_raw_run042433_0300_df-s03-d1_dw_0_20260203T203609.hdf5
JobscriptExit code0
Real time1h (5005s)
CPU time1h (4366s = 87%)
Max RSS bytes3417366528 (3259 MiB)
Outputting started2026-02-04 01:08:49
Output files
Finished2026-02-04 01:11:23
Saved logsjustin-logs:297921.3-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

point ptr... 0x13718fb0, 0x13718e20
Caught exception in looking up space point ptr... 0x13718e20, 0x13788ac0
Caught exception in looking up space point ptr... 0x13789290, 0x13788f70
Caught exception in looking up space point ptr... 0x13788f70, 0x13718fb0
Caught exception in looking up space point ptr... 0x4b3f2d40, 0x4b3f2850
Caught exception in looking up space point ptr... 0x4b3f2850, 0x4b3f2d40
Caught exception in looking up space point ptr... 0x36cade80, 0x2a04dcf0
Caught exception in looking up space point ptr... 0x29f18360, 0x36cade80
Caught exception in looking up space point ptr... 0x51ba7dd0, 0x51bad7b0
Caught exception in looking up space point ptr... 0x51bad7b0, 0x51bad620
Caught exception in looking up space point ptr... 0x64da6730, 0x64da65a0
Caught exception in looking up space point ptr... 0x64da65a0, 0x64da6730
Caught exception in looking up space point ptr... 0x603cd980, 0x603cd6e0
Caught exception in looking up space point ptr... 0x603cd6e0, 0x603cd980
Caught exception in looking up space point ptr... 0x60a466a0, 0x8398d910
Caught exception in looking up space point ptr... 0x8398d910, 0x60a466a0
Caught exception in looking up space point ptr... 0x4b3fc080, 0x4b3ef230
Caught exception in looking up space point ptr... 0x4b3ef230, 0x4b3fc080
Caught exception in looking up space point ptr... 0x3b8ca1b0, 0x3b8c9eb0
Caught exception in looking up space point ptr... 0x3b8c9eb0, 0x3b8ca1b0
Caught exception in looking up space point ptr... 0x9026560, 0x9025c00
Caught exception in looking up space point ptr... 0x9025c00, 0x9025e60
Caught exception in looking up space point ptr... 0x348af140, 0x348af040
Caught exception in looking up space point ptr... 0x348af040, 0x348af240
Caught exception in looking up space point ptr... 0x1154f720, 0x1154ef50
Caught exception in looking up space point ptr... 0x1154ef50, 0x1154f720
Caught exception in looking up space point ptr... 0x1154f720, 0x1154f400
Caught exception in looking up space point ptr... 0x1154f400, 0x1154f720
Caught exception in looking up space point ptr... 0x1154f720, 0x1154f270
Caught exception in looking up space point ptr... 0x1154f270, 0x1154f720
Caught exception in looking up space point ptr... 0x1154f720, 0x1154f590
Caught exception in looking up space point ptr... 0x1154f590, 0x1154f720
Caught exception in looking up space point ptr... 0x44983840, 0x449831d0
Caught exception in looking up space point ptr... 0x449831d0, 0x44983300
Caught exception in looking up space point ptr... 0xd9bf120, 0xd9bf3c0
Caught exception in looking up space point ptr... 0xd9bf3c0, 0xd9bf120
Caught exception in looking up space point ptr... 0x29e3ca10, 0x29e3bb00
Caught exception in looking up space point ptr... 0x29e3bb00, 0x29e3ca10
Caught exception in looking up space point ptr... 0x43a80000, 0x6b4f3960
Caught exception in looking up space point ptr... 0x6b4f3960, 0x43a7fe20
Caught exception in looking up space point ptr... 0x6e81bd90, 0x6e81ba70
Caught exception in looking up space point ptr... 0x6e81ba70, 0x6e81bc00
Caught exception in looking up space point ptr... 0x118a82f0, 0x118a7fd0
Caught exception in looking up space point ptr... 0x118a7fd0, 0x118a82f0
Caught exception in looking up space point ptr... 0xe76e6c0, 0xe76db20
Caught exception in looking up space point ptr... 0xe76db20, 0xe76e6c0
Caught exception in looking up space point ptr... 0x7f87ca40, 0x39395d00
Caught exception in looking up space point ptr... 0x39395d00, 0x7f87cb40
Caught exception in looking up space point ptr... 0x7f87cb40, 0x39395e50
Caught exception in looking up space point ptr... 0x466572e0, 0x46657790
Caught exception in looking up space point ptr... 0x46657790, 0x466572e0
Caught exception in looking up space point ptr... 0x4c5bbe50, 0x4c5bbd00
Caught exception in looking up space point ptr... 0x4c5bbd00, 0x4c5bbe50
Caught exception in looking up space point ptr... 0x83965370, 0x83965500
Caught exception in looking up space point ptr... 0x83965500, 0x83965050
Caught exception in looking up space point ptr... 0x4a2dfb10, 0xdadf760
Caught exception in looking up space point ptr... 0xdadf560, 0x4a2dfb10
Caught exception in looking up space point ptr... 0x50b75c10, 0x50b74bd0
Caught exception in looking up space point ptr... 0x50b74bd0, 0x50b76b20
Caught exception in looking up space point ptr... 0x46653dd0, 0x46653ed0
Caught exception in looking up space point ptr... 0x46653ed0, 0x46653dd0
Caught exception in looking up space point ptr... 0x83965b40, 0x839659b0
Caught exception in looking up space point ptr... 0x839659b0, 0x83965b40
Caught exception in looking up space point ptr... 0x151cdf70, 0x151cdde0
Caught exception in looking up space point ptr... 0x151cdde0, 0x151ce100
Caught exception in looking up space point ptr... 0x64b604b0, 0x5a7bd600
Caught exception in looking up space point ptr... 0x5a7bd600, 0x64b3d820
++++>>>> total num hits: 147010, num free: 141632
04-Feb-2026 02:08:33 CET  Closed output file "np02vd_raw_run042433_0300_df-s03-d1_dw_0_20260203T203609_reco_stage0_20260204T010833_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         196.788       255.86        300.443       263.016       30.9898        19     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                        7.002e-05    0.000135958   0.000409051   0.00010832    7.36029e-05      19     
produce:tpcrawdecoder:PDVDTPCReader                24.0155       30.4408       42.6998       28.2067       6.16606        19     
produce:triggerrawdecoder:PDVDTriggerReader4      0.311312      0.348292      0.430502      0.322479      0.0401562       19     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.0104259     0.0112432     0.0163255     0.0106611    0.00162728       19     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.0103223     0.0107018     0.0117299     0.0106236    0.000284384      19     
produce:beamevent:BeamEvent                      0.00011326    0.000220031   0.00140069    0.000160071   0.000279037      19     
produce:pdvddaphne:DAPHNEReaderPDVD                3.97099       5.13457       6.40883       5.22776      0.721008        19     
produce:ophit:OpHitFinder                         0.0394722     0.0500575     0.0593317     0.0493541    0.00568268       19     
produce:wclsdatavd:WireCellToolkit                 129.982       176.423       208.797       178.808       19.861         19     
produce:gaushit:GausHitFinder                      1.25174       2.26989       3.37703       2.26361      0.548542        19     
produce:nhitsfilter:NumberOfHitsFilter           0.000445581   0.000574129   0.000932682   0.000540841    0.0001224       19     
produce:reco3d:SpacePointSolver                    13.9531       26.5641       40.5091       26.152        8.52001        19     
produce:hitpdune:DisambigFromSpacePoints          0.136879      0.337364      0.596581       0.2974       0.121238        19     
produce:cluster3d:Cluster3D                        2.19458       5.88682       23.4102       4.71885       4.60227        19     
[art]:TriggerResults:TriggerResultInserter        2.378e-05    3.96827e-05   0.00012696     3.144e-05    2.41322e-05      19     
end_path:out1:RootOutput                          6.97e-06     1.21068e-05    3.69e-05      1.013e-05    6.42419e-06      19     
end_path:out1:RootOutput(write)                    5.8094        8.33182       10.3222       8.36303       1.22782        19     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5371.13 MB
  Peak resident set size usage (VmHWM): 3417.37 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042433_0300_df-s03-d1_dw_0_20260203T203609_reco_stage0_20260204T010833_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042433_0300_df-s03-d1_dw_0_20260203T203609_reco_stage0_20260204T010833_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042433_0300_df-s03-d1_dw_0_20260203T203609_reco_stage0_20260204T010833_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770162340.0,
    "core.end_time": 1770167317.0,
    "core.events": [
      34209,
      34215,
      34221,
      34227,
      34233,
      34239,
      34245,
      34251,
      34257,
      34263,
      34269,
      34275,
      34281,
      34287,
      34293,
      34299,
      34305,
      34311,
      34317
    ],
    "core.event_count": 19,
    "core.first_event_number": 34209,
    "core.last_event_number": 34317,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42433
    ],
    "core.runs_subruns": [
      4243300001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042433_0300_df-s03-d1_dw_0_20260203T203609.hdf5"
    }
  ]
}
justIN time: 2026-02-04 04:45:46 UTC       justIN version: 01.06.00