justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 303651.189@dunegpschedd01.fnal.gov

Jobsub ID303651.189@dunegpschedd01.fnal.gov
Workflow ID12717
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:50:24
SiteCERN
EntryCMSHTPC_T2_CH_CERN_ce512
Last heartbeat2026-02-04 00:46:27
From worker nodeHostnameb9g24p4713.cern.ch
cpuinfoAMD EPYC 7302 16-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit343800 (95 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 23:24:01
Input filesvd-protodune:np02vd_raw_run042421_0142_df-s03-d1_dw_0_20260203T144748.hdf5
JobscriptExit code0
Real time1h (4769s)
CPU time1h (4635s = 97%)
Max RSS bytes11826581504 (11278 MiB)
Outputting started2026-02-04 00:43:34
Output files
Finished2026-02-04 00:46:27
Saved logsjustin-logs:303651.189-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

ught exception in looking up space point ptr... 0x3e09d3f0, 0x29b7dd10
Caught exception in looking up space point ptr... 0x2be27d50, 0x3354a5f0
Caught exception in looking up space point ptr... 0x3354a5f0, 0x2be27d50
Caught exception in looking up space point ptr... 0x3899f010, 0x3899f1b0
Caught exception in looking up space point ptr... 0x3899f1b0, 0x389a36d0
Caught exception in looking up space point ptr... 0x95399c30, 0x95399970
Caught exception in looking up space point ptr... 0x95399970, 0x95399ac0
Caught exception in looking up space point ptr... 0x12b0d8e0, 0x12b0d5c0
Caught exception in looking up space point ptr... 0x12b092f0, 0x12b0d8e0
Caught exception in looking up space point ptr... 0x38fd2ff0, 0x38fd2e60
Caught exception in looking up space point ptr... 0x38fd2e60, 0x38fd2ff0
Caught exception in looking up space point ptr... 0x12afc280, 0x95597af0
Caught exception in looking up space point ptr... 0x95597af0, 0x12afbb70
Caught exception in looking up space point ptr... 0xcf62d00, 0xcf62a10
Caught exception in looking up space point ptr... 0xcf62a10, 0xcf62910
Caught exception in looking up space point ptr... 0x95221860, 0x952213a0
Caught exception in looking up space point ptr... 0x95221ac0, 0x95221860
Caught exception in looking up space point ptr... 0x3e269390, 0x5909c020
Caught exception in looking up space point ptr... 0x5909c020, 0x3e269a30
Caught exception in looking up space point ptr... 0x955a4b60, 0x12af5460
Caught exception in looking up space point ptr... 0x12af5460, 0x955a4b60
Caught exception in looking up space point ptr... 0x955a4b60, 0x955a6510
Caught exception in looking up space point ptr... 0x955a6510, 0x955a4b60
Caught exception in looking up space point ptr... 0xcb1aebf0, 0xcb1aea60
Caught exception in looking up space point ptr... 0xcb1b24d0, 0xcb1aebf0
Caught exception in looking up space point ptr... 0x11f82ce0, 0x11f88490
Caught exception in looking up space point ptr... 0x11f88490, 0x11f82ce0
Caught exception in looking up space point ptr... 0x3a12e440, 0x51eb7f30
Caught exception in looking up space point ptr... 0x51eb7f30, 0x3a12e440
Caught exception in looking up space point ptr... 0x11630f460, 0x10a8d1d0
Caught exception in looking up space point ptr... 0x33947550, 0x11630f460
Caught exception in looking up space point ptr... 0x14bc8cb0, 0x92ff920
Caught exception in looking up space point ptr... 0x92ff920, 0x14bc8cb0
Caught exception in looking up space point ptr... 0xc0a137a0, 0xc0a132f0
Caught exception in looking up space point ptr... 0xc0a132f0, 0xc0a137a0
Caught exception in looking up space point ptr... 0x34aaaf30, 0x2d4e4990
Caught exception in looking up space point ptr... 0x2d4e4990, 0x34aaaf30
Caught exception in looking up space point ptr... 0x328cf7b0, 0x4046d7f0
Caught exception in looking up space point ptr... 0x4046d7f0, 0x328cf7b0
Caught exception in looking up space point ptr... 0x32749690, 0x51eef630
Caught exception in looking up space point ptr... 0x51eef630, 0x32749690
Caught exception in looking up space point ptr... 0x32749690, 0x51fc52a0
Caught exception in looking up space point ptr... 0x51fc52a0, 0x32749690
Caught exception in looking up space point ptr... 0x28fcda80, 0x8662dc0
Caught exception in looking up space point ptr... 0x8662dc0, 0x28fcda80
Caught exception in looking up space point ptr... 0x2c758810, 0x2c758620
Caught exception in looking up space point ptr... 0x2c758620, 0x2c758810
Caught exception in looking up space point ptr... 0x33c66620, 0x33c6a110
Caught exception in looking up space point ptr... 0x33c6a110, 0x33c6f480
Caught exception in looking up space point ptr... 0x569579d0, 0x11369c530
Caught exception in looking up space point ptr... 0x11369c530, 0x28e72520
Caught exception in looking up space point ptr... 0x9fc5e930, 0x9fc64800
Caught exception in looking up space point ptr... 0x9fc64800, 0x9fc64950
Caught exception in looking up space point ptr... 0x1396c8a0, 0x1396c740
Caught exception in looking up space point ptr... 0x1396c740, 0x1396c8a0
Caught exception in looking up space point ptr... 0x95583a80, 0x95582620
Caught exception in looking up space point ptr... 0x95582620, 0x95583a80
Caught exception in looking up space point ptr... 0x4d72a260, 0x4d72a0d0
Caught exception in looking up space point ptr... 0x4d72a0d0, 0x2aec3ac0
Caught exception in looking up space point ptr... 0xa56410d0, 0xa5640da0
Caught exception in looking up space point ptr... 0xa5640da0, 0xa56410d0
Caught exception in looking up space point ptr... 0xa5661ac0, 0xf7aea9c0
Caught exception in looking up space point ptr... 0xf7aea9c0, 0xa5661ac0
Caught exception in looking up space point ptr... 0x59441b60, 0x594416a0
Caught exception in looking up space point ptr... 0x594416a0, 0x59441b60
++++>>>> total num hits: 74707, num free: 72472
04-Feb-2026 01:43:24 CET  Closed output file "np02vd_raw_run042421_0142_df-s03-d1_dw_0_20260203T144748_reco_stage0_20260204T004324_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         77.1116       174.016       897.233       90.5775       207.276        27     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       5.0091e-05    8.90051e-05   0.000281915   6.9582e-05    5.24827e-05      27     
produce:tpcrawdecoder:PDVDTPCReader                3.61924       4.47767       11.7553       4.19276       1.45813        27     
produce:triggerrawdecoder:PDVDTriggerReader4      0.0108156     0.029077      0.224576      0.0153228     0.040944        27     
produce:timingrawdecoder:PDHDTimingRawDecoder    0.000405688   0.000598926   0.00109542    0.00052322    0.000173554      27     
produce:ctbrawdecoder:PDHDCTBRawDecoder          0.00171075     0.0020594    0.00371238    0.00193869    0.000395672      27     
produce:beamevent:BeamEvent                      5.4291e-05    0.000118474   0.00102336    6.7522e-05    0.000181361      27     
produce:pdvddaphne:DAPHNEReaderPDVD                2.10669       2.44092       2.96396       2.42671      0.196937        27     
produce:ophit:OpHitFinder                         0.0230878     0.0330157     0.0424622     0.0337122    0.00453456       27     
produce:wclsdatavd:WireCellToolkit                 61.0191       65.8859       72.1257       65.7234       2.55971        27     
produce:gaushit:GausHitFinder                     0.535222       1.79449       13.2515      0.876296       2.60303        27     
produce:nhitsfilter:NumberOfHitsFilter           0.000148573   0.000389263   0.00101645    0.000315606   0.000181821      27     
produce:reco3d:SpacePointSolver                    4.47598       16.4203       66.4234       12.011        14.8859        27     
produce:hitpdune:DisambigFromSpacePoints          0.0522692     0.379739       2.37101      0.107946      0.629056        27     
produce:cluster3d:Cluster3D                       0.655573       77.0907       721.568       1.85462       184.893        27     
[art]:TriggerResults:TriggerResultInserter        2.134e-05    3.1928e-05    0.000102832    2.684e-05    1.57991e-05      27     
end_path:out1:RootOutput                          4.48e-06     9.22393e-06   2.4781e-05     8.55e-06     3.30156e-06      27     
end_path:out1:RootOutput(write)                    3.14701       5.43578       22.4785       3.50495       4.66934        27     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 14030.2 MB
  Peak resident set size usage (VmHWM): 11826.6 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042421_0142_df-s03-d1_dw_0_20260203T144748_reco_stage0_20260204T004324_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042421_0142_df-s03-d1_dw_0_20260203T144748_reco_stage0_20260204T004324_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042421_0142_df-s03-d1_dw_0_20260203T144748_reco_stage0_20260204T004324_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770161054.0,
    "core.end_time": 1770165806.0,
    "core.events": [
      23073,
      23079,
      23085,
      23091,
      23097,
      23103,
      23109,
      23115,
      23121,
      23127,
      23133,
      23139,
      23145,
      23151,
      23157,
      23163,
      23169,
      23175,
      23181,
      23187,
      23193,
      23199,
      23205,
      23211,
      23217,
      23223,
      23229
    ],
    "core.event_count": 27,
    "core.first_event_number": 23073,
    "core.last_event_number": 23229,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42421
    ],
    "core.runs_subruns": [
      4242100001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042421_0142_df-s03-d1_dw_0_20260203T144748.hdf5"
    }
  ]
}
justIN time: 2026-02-04 06:07:40 UTC       justIN version: 01.06.00