justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 303652.178@dunegpschedd01.fnal.gov

Jobsub ID303652.178@dunegpschedd01.fnal.gov
Workflow ID12719
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:50:26
SiteCERN
EntryCMSHTPC_T2_CH_CERN_ce510
Last heartbeat2026-02-04 00:26:02
From worker nodeHostnameb9p09p3861.cern.ch
cpuinfoAMD EPYC 7543 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit343800 (95 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 23:34:24
Input filesvd-protodune:np02vd_raw_run042433_0349_df-s03-d1_dw_0_20260203T205445.hdf5
JobscriptExit code0
Real time49m (2948s)
CPU time47m (2834s = 96%)
Max RSS bytes3400433664 (3242 MiB)
Outputting started2026-02-04 00:23:33
Output files
Finished2026-02-04 00:26:02
Saved logsjustin-logs:303652.178-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

ce point ptr... 0x9cfcd20, 0x9d059b0
Caught exception in looking up space point ptr... 0x9d059b0, 0x9cf6c90
Caught exception in looking up space point ptr... 0xf068de0, 0xf0724a0
Caught exception in looking up space point ptr... 0xf0724a0, 0xf069640
Caught exception in looking up space point ptr... 0x481fcc30, 0x400c02f0
Caught exception in looking up space point ptr... 0x400c02f0, 0x157d7a80
Caught exception in looking up space point ptr... 0x45357270, 0x4562c6d0
Caught exception in looking up space point ptr... 0x4562c6d0, 0x2dede550
Caught exception in looking up space point ptr... 0x196fb9d0, 0x295ff910
Caught exception in looking up space point ptr... 0x295ff910, 0x295ff7c0
Caught exception in looking up space point ptr... 0x5b5986e0, 0x5012a320
Caught exception in looking up space point ptr... 0x5012a320, 0x630fe360
Caught exception in looking up space point ptr... 0x6e4c9aa0, 0x64351d40
Caught exception in looking up space point ptr... 0x64351d40, 0x427d0900
Caught exception in looking up space point ptr... 0x6965ef10, 0x6965e300
Caught exception in looking up space point ptr... 0x69660950, 0x6965ef10
Caught exception in looking up space point ptr... 0xdb373f0, 0xdb3a280
Caught exception in looking up space point ptr... 0xdb3a280, 0xdb41b00
Caught exception in looking up space point ptr... 0x37202990, 0x371fdad0
Caught exception in looking up space point ptr... 0x371fde20, 0x37202990
Caught exception in looking up space point ptr... 0x39f63be0, 0x7caa6440
Caught exception in looking up space point ptr... 0x7caa6440, 0x39f63be0
Caught exception in looking up space point ptr... 0x60e04d70, 0x60e042d0
Caught exception in looking up space point ptr... 0x60e042d0, 0x60e04d70
Caught exception in looking up space point ptr... 0x60d7f040, 0x60d7ed20
Caught exception in looking up space point ptr... 0x60d7ed20, 0x60d7f040
Caught exception in looking up space point ptr... 0x62f47de0, 0x5007efa0
Caught exception in looking up space point ptr... 0x5007efa0, 0x4d250c20
Caught exception in looking up space point ptr... 0xdb3b6d0, 0xdb3b3d0
Caught exception in looking up space point ptr... 0xdb3b3d0, 0xdb3b6d0
Caught exception in looking up space point ptr... 0x4818d6e0, 0x3d188150
Caught exception in looking up space point ptr... 0x3d188150, 0x4818d6e0
Caught exception in looking up space point ptr... 0x80ff2a50, 0x80ff38d0
Caught exception in looking up space point ptr... 0x80ff38d0, 0x62f74e40
Caught exception in looking up space point ptr... 0x62f74e40, 0x62f69bb0
Caught exception in looking up space point ptr... 0x62f69bb0, 0x5b57eed0
Caught exception in looking up space point ptr... 0x666f0370, 0x80ff2a50
Caught exception in looking up space point ptr... 0x6968fe00, 0x6968b260
Caught exception in looking up space point ptr... 0x6968b260, 0x6968fe00
Caught exception in looking up space point ptr... 0x37219d20, 0x109d0690
Caught exception in looking up space point ptr... 0x109d0690, 0x37219d20
Caught exception in looking up space point ptr... 0x5e07e9f0, 0x5e07c990
Caught exception in looking up space point ptr... 0x5e07c990, 0x5e07e9f0
Caught exception in looking up space point ptr... 0x519466d0, 0x51946820
Caught exception in looking up space point ptr... 0x51946820, 0x519466d0
Caught exception in looking up space point ptr... 0x3feefe90, 0x1338a930
Caught exception in looking up space point ptr... 0x1338a930, 0x3cd14d10
Caught exception in looking up space point ptr... 0x127c8300, 0x127c56a0
Caught exception in looking up space point ptr... 0x127c56a0, 0x19802510
Caught exception in looking up space point ptr... 0x66d99040, 0x66d98d20
Caught exception in looking up space point ptr... 0x66d98d20, 0x66d98eb0
Caught exception in looking up space point ptr... 0x19736540, 0x19730e90
Caught exception in looking up space point ptr... 0x19730e90, 0x19735710
Caught exception in looking up space point ptr... 0x4fd15e30, 0x853e0370
Caught exception in looking up space point ptr... 0x853e0370, 0x4fd15890
Caught exception in looking up space point ptr... 0x4d181600, 0x1225fec0
Caught exception in looking up space point ptr... 0x1225fec0, 0x4d181600
Caught exception in looking up space point ptr... 0x13a0d530, 0x13a0d850
Caught exception in looking up space point ptr... 0x13a0d850, 0x13a0db70
Caught exception in looking up space point ptr... 0x115f1020, 0x115f3a40
Caught exception in looking up space point ptr... 0x115f3a40, 0x115f1020
Caught exception in looking up space point ptr... 0x31471d70, 0x2a8f1ba0
Caught exception in looking up space point ptr... 0x2a8f1ba0, 0x4554f190
Caught exception in looking up space point ptr... 0x66d87f30, 0x66d87930
Caught exception in looking up space point ptr... 0x66d87930, 0x66d863d0
Caught exception in looking up space point ptr... 0x7bf78350, 0x35396040
Caught exception in looking up space point ptr... 0x35396040, 0x7bf78350
++++>>>> total num hits: 110296, num free: 107348
04-Feb-2026 01:23:24 CET  Closed output file "np02vd_raw_run042433_0349_df-s03-d1_dw_0_20260203T205445_reco_stage0_20260204T002324_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         121.977       151.345       179.293       150.818       14.9259        19     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       4.2891e-05    7.87517e-05   0.000153093   7.3451e-05    2.92658e-05      19     
produce:tpcrawdecoder:PDVDTPCReader                6.20735       7.22321       8.3704        7.24858      0.454228        19     
produce:triggerrawdecoder:PDVDTriggerReader4      0.0113379     0.0234719     0.0580066     0.0196814     0.0128239       19     
produce:timingrawdecoder:PDHDTimingRawDecoder    0.000427487   0.000522637   0.000716311   0.000503078   6.37473e-05      19     
produce:ctbrawdecoder:PDHDCTBRawDecoder          0.000392206   0.000527951   0.000714451   0.000513238   7.78654e-05      19     
produce:beamevent:BeamEvent                      5.5971e-05    0.000134776   0.000950245   8.7101e-05    0.000193014      19     
produce:pdvddaphne:DAPHNEReaderPDVD                2.42391       2.94493       3.69566       2.97723      0.324973        19     
produce:ophit:OpHitFinder                         0.0228437     0.0365444     0.0452807     0.0386319    0.00593581       19     
produce:wclsdatavd:WireCellToolkit                 95.5998       111.453       136.235       110.12        9.53296        19     
produce:gaushit:GausHitFinder                     0.937939       1.45662       2.40942       1.37155      0.417617        19     
produce:nhitsfilter:NumberOfHitsFilter           0.000225803   0.000386437   0.000747041   0.000369345   0.000120317      19     
produce:reco3d:SpacePointSolver                    9.04648       19.9382       33.7911        17.07        7.03521        19     
produce:hitpdune:DisambigFromSpacePoints          0.0808553     0.197793       0.41479      0.177104      0.0775312       19     
produce:cluster3d:Cluster3D                        1.17402        2.661        5.64782       2.59534      0.981783        19     
[art]:TriggerResults:TriggerResultInserter       2.3151e-05    4.18203e-05   0.000222274   2.9191e-05    4.36363e-05      19     
end_path:out1:RootOutput                          5.01e-06     8.86174e-06   3.3331e-05      7.7e-06     5.90917e-06      19     
end_path:out1:RootOutput(write)                    4.35096       5.38009       6.65847       5.22678       0.73816        19     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5369.57 MB
  Peak resident set size usage (VmHWM): 3400.43 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042433_0349_df-s03-d1_dw_0_20260203T205445_reco_stage0_20260204T002324_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042433_0349_df-s03-d1_dw_0_20260203T205445_reco_stage0_20260204T002324_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042433_0349_df-s03-d1_dw_0_20260203T205445_reco_stage0_20260204T002324_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770161676.0,
    "core.end_time": 1770164605.0,
    "core.events": [
      39795,
      39801,
      39807,
      39813,
      39819,
      39825,
      39831,
      39837,
      39843,
      39849,
      39855,
      39861,
      39867,
      39873,
      39879,
      39885,
      39891,
      39897,
      39903
    ],
    "core.event_count": 19,
    "core.first_event_number": 39795,
    "core.last_event_number": 39903,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42433
    ],
    "core.runs_subruns": [
      4243300001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042433_0349_df-s03-d1_dw_0_20260203T205445.hdf5"
    }
  ]
}
justIN time: 2026-02-04 05:56:26 UTC       justIN version: 01.06.00