justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 297921.31@dunegpschedd02.fnal.gov

Jobsub ID297921.31@dunegpschedd02.fnal.gov
Workflow ID12719
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:58:26
SiteIT_CNAF
EntryCMSHTPC_T1_IT_CNAF_condor_ce02
Last heartbeat2026-02-04 01:27:12
From worker nodeHostnamewn-s04-06-11-04.cr.cnaf.infn.it
cpuinfoAMD EPYC 7351 16-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 23:47:04
Input filesvd-protodune:np02vd_raw_run042433_0297_df-s02-d1_dw_0_20260203T203500.hdf5
JobscriptExit code0
Real time1h (5845s)
CPU time1h (5181s = 88%)
Max RSS bytes3601760256 (3434 MiB)
Outputting started2026-02-04 01:24:32
Output files
Finished2026-02-04 01:27:12
Saved logsjustin-logs:297921.31-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

... 0x36e949c0, 0x38de5150
Caught exception in looking up space point ptr... 0x60e7aa10, 0x30a7ea90
Caught exception in looking up space point ptr... 0x39115ab0, 0x60e7aa10
Caught exception in looking up space point ptr... 0x3d0016b0, 0x3d001750
Caught exception in looking up space point ptr... 0x3d001750, 0x3d0016b0
Caught exception in looking up space point ptr... 0x4e63bd60, 0x38fde920
Caught exception in looking up space point ptr... 0x38fde920, 0x4e63bc60
Caught exception in looking up space point ptr... 0x3dddeb40, 0x3ddde9b0
Caught exception in looking up space point ptr... 0x3ddde9b0, 0x38633830
Caught exception in looking up space point ptr... 0x49211110, 0x49210b90
Caught exception in looking up space point ptr... 0x49210a60, 0x406e93f0
Caught exception in looking up space point ptr... 0x406e93f0, 0x3a588fc0
Caught exception in looking up space point ptr... 0x3f0eae70, 0x49211110
Caught exception in looking up space point ptr... 0x3d57fb50, 0x387a0990
Caught exception in looking up space point ptr... 0x387a0990, 0x3d57fb50
Caught exception in looking up space point ptr... 0x661bc1d0, 0x3ddcd260
Caught exception in looking up space point ptr... 0x3ddcd260, 0x393f10c0
Caught exception in looking up space point ptr... 0x6195e0f0, 0x2a1370a0
Caught exception in looking up space point ptr... 0x2a1370a0, 0x6195e0f0
Caught exception in looking up space point ptr... 0x80dcdf30, 0x5a2375c0
Caught exception in looking up space point ptr... 0x80dce060, 0x80dcdf30
Caught exception in looking up space point ptr... 0x18a5aab0, 0x613d3cb0
Caught exception in looking up space point ptr... 0x66cd6620, 0x18a5aab0
Caught exception in looking up space point ptr... 0x3d5ccb80, 0x3d5cdae0
Caught exception in looking up space point ptr... 0x3d5cdae0, 0x3d5ccb80
Caught exception in looking up space point ptr... 0x3d6563d0, 0x3b043aa0
Caught exception in looking up space point ptr... 0x3b043aa0, 0x3d6563d0
Caught exception in looking up space point ptr... 0x75724480, 0x75724f70
Caught exception in looking up space point ptr... 0x75724f70, 0x75724480
Caught exception in looking up space point ptr... 0x3d87d9a0, 0x1063ba00
Caught exception in looking up space point ptr... 0x1063ba00, 0x3d87d9a0
Caught exception in looking up space point ptr... 0x657c1e0, 0x5395da70
Caught exception in looking up space point ptr... 0x5395da70, 0x657c570
Caught exception in looking up space point ptr... 0x89fe3200, 0x2a61b980
Caught exception in looking up space point ptr... 0x2a61b980, 0x89fe3200
Caught exception in looking up space point ptr... 0x3b19b480, 0x39534e30
Caught exception in looking up space point ptr... 0x39534e30, 0x3b19b480
Caught exception in looking up space point ptr... 0x6274adf0, 0x3aaeab80
Caught exception in looking up space point ptr... 0x3aaeab80, 0x6274adf0
Caught exception in looking up space point ptr... 0x15b56a10, 0x15b56d30
Caught exception in looking up space point ptr... 0x15b56d30, 0x15b56a10
Caught exception in looking up space point ptr... 0x66a960d0, 0x66a96220
Caught exception in looking up space point ptr... 0x66a96220, 0x66a960d0
Caught exception in looking up space point ptr... 0x18539590, 0x32c80350
Caught exception in looking up space point ptr... 0x32c80350, 0x18539590
Caught exception in looking up space point ptr... 0x18539590, 0x32c804e0
Caught exception in looking up space point ptr... 0x32c804e0, 0x18539590
Caught exception in looking up space point ptr... 0x13dc6e40, 0x559532a0
Caught exception in looking up space point ptr... 0x559532a0, 0x13dc6e40
Caught exception in looking up space point ptr... 0x390a70b0, 0x390a7350
Caught exception in looking up space point ptr... 0x390a7350, 0x390a70b0
Caught exception in looking up space point ptr... 0x378d1e50, 0x378d1cf0
Caught exception in looking up space point ptr... 0x39038f60, 0x378d1e50
Caught exception in looking up space point ptr... 0x61cee6f0, 0x438ad620
Caught exception in looking up space point ptr... 0x438ad620, 0x61cee6f0
Caught exception in looking up space point ptr... 0x663ee110, 0x663ede60
Caught exception in looking up space point ptr... 0x663ede60, 0x663ee2d0
Caught exception in looking up space point ptr... 0x60929ea0, 0x60929d10
Caught exception in looking up space point ptr... 0x60929d10, 0x60929ea0
Caught exception in looking up space point ptr... 0x15b57050, 0x15b57370
Caught exception in looking up space point ptr... 0x15b57370, 0x15b57050
Caught exception in looking up space point ptr... 0x15b576e0, 0x15b57500
Caught exception in looking up space point ptr... 0x15b57500, 0x15b57a00
Caught exception in looking up space point ptr... 0x3b0854c0, 0x3b084e80
Caught exception in looking up space point ptr... 0x3b084e80, 0x3b0854c0
Caught exception in looking up space point ptr... 0x3e638520, 0x504d5eb0
Caught exception in looking up space point ptr... 0x504d5eb0, 0x3e638520
++++>>>> total num hits: 80047, num free: 76401
04-Feb-2026 02:24:18 CET  Closed output file "np02vd_raw_run042433_0297_df-s02-d1_dw_0_20260203T203500_reco_stage0_20260204T012418_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         222.695       301.368       403.482       304.435       50.1298        19     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                        6.374e-05    0.000125905   0.000322031   0.00010429    5.48203e-05      19     
produce:tpcrawdecoder:PDVDTPCReader                20.3119       31.3106       40.8887       30.8967       5.4831         19     
produce:triggerrawdecoder:PDVDTriggerReader4      0.330103      0.401642      0.525365      0.393332      0.0526543       19     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.0102681     0.0150285     0.0609417     0.0108552     0.0111998       19     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.0101782     0.0106808     0.0115783     0.0106656    0.000287516      19     
produce:beamevent:BeamEvent                       8.267e-05    0.000247222   0.00155314    0.00018048    0.000310016      19     
produce:pdvddaphne:DAPHNEReaderPDVD                4.23732       5.45355       7.73463       5.37143      0.807462        19     
produce:ophit:OpHitFinder                         0.0354055     0.0519013     0.066012      0.0556528    0.00986537       19     
produce:wclsdatavd:WireCellToolkit                 151.336       179.343       228.748       175.436       17.7981        19     
produce:gaushit:GausHitFinder                      1.61155       2.76141       4.25561       2.58403      0.735747        19     
produce:nhitsfilter:NumberOfHitsFilter           0.000414591   0.000547071   0.000944012   0.000495591   0.000132485      19     
produce:reco3d:SpacePointSolver                    13.7159       20.3911       35.3939       20.2264       5.01475        19     
produce:hitpdune:DisambigFromSpacePoints          0.231451       0.42194      0.611252      0.430296      0.101961        19     
produce:cluster3d:Cluster3D                        2.79298       51.3931       94.1935       73.9587       39.4193        19     
[art]:TriggerResults:TriggerResultInserter        2.943e-05    4.85392e-05   0.000135131    4.201e-05    2.4218e-05       19     
end_path:out1:RootOutput                          7.09e-06      0.0004621    0.00855392     1.128e-05    0.00190727       19     
end_path:out1:RootOutput(write)                    6.41214       9.74054       12.2598       9.51969       1.64582        19     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5542.65 MB
  Peak resident set size usage (VmHWM): 3601.76 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042433_0297_df-s02-d1_dw_0_20260203T203500_reco_stage0_20260204T012418_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042433_0297_df-s02-d1_dw_0_20260203T203500_reco_stage0_20260204T012418_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042433_0297_df-s02-d1_dw_0_20260203T203500_reco_stage0_20260204T012418_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770162441.0,
    "core.end_time": 1770168262.0,
    "core.events": [
      33865,
      33871,
      33877,
      33883,
      33889,
      33895,
      33901,
      33907,
      33913,
      33919,
      33925,
      33931,
      33937,
      33943,
      33949,
      33955,
      33961,
      33967,
      33973
    ],
    "core.event_count": 19,
    "core.first_event_number": 33865,
    "core.last_event_number": 33973,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42433
    ],
    "core.runs_subruns": [
      4243300001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042433_0297_df-s02-d1_dw_0_20260203T203500.hdf5"
    }
  ]
}
justIN time: 2026-02-04 04:47:28 UTC       justIN version: 01.06.00