justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 303652.19@dunegpschedd01.fnal.gov

Jobsub ID303652.19@dunegpschedd01.fnal.gov
Workflow ID12719
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:50:26
SiteCERN
EntryCMSHTPC_T2_CH_CERN_ce503
Last heartbeat2026-02-04 00:10:38
From worker nodeHostnameb9p05p0564.cern.ch
cpuinfoAMD EPYC 7543 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit343800 (95 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 23:27:22
Input filesvd-protodune:np02vd_raw_run042433_0335_df-s03-d0_dw_0_20260203T204926.hdf5
JobscriptExit code0
Real time40m (2453s)
CPU time39m (2369s = 96%)
Max RSS bytes3416367104 (3258 MiB)
Outputting started2026-02-04 00:08:16
Output files
Finished2026-02-04 00:10:38
Saved logsjustin-logs:303652.19-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

e point ptr... 0x3afe3bf0, 0x84035c00
Caught exception in looking up space point ptr... 0x84035c00, 0x46fa2f40
Caught exception in looking up space point ptr... 0x46fa2f40, 0x38295780
Caught exception in looking up space point ptr... 0x37e5a9f0, 0x3b34f250
Caught exception in looking up space point ptr... 0x3b34f250, 0x3b34f3a0
Caught exception in looking up space point ptr... 0x610c070, 0x368a18d0
Caught exception in looking up space point ptr... 0x368a18d0, 0x610c070
Caught exception in looking up space point ptr... 0x3a5881b0, 0x3a588350
Caught exception in looking up space point ptr... 0x3a588350, 0x66c21ac0
Caught exception in looking up space point ptr... 0x3c9cecf0, 0x3c9cfd50
Caught exception in looking up space point ptr... 0x3c9cfd50, 0x3c9cecf0
Caught exception in looking up space point ptr... 0x37c8cd50, 0x50af08d0
Caught exception in looking up space point ptr... 0x50af08d0, 0x6be4a750
Caught exception in looking up space point ptr... 0x7fd6090, 0x7fd62f0
Caught exception in looking up space point ptr... 0x7fd62f0, 0x7fd6090
Caught exception in looking up space point ptr... 0x7fd6090, 0x7fd6470
Caught exception in looking up space point ptr... 0x7fd6470, 0x7fd6090
Caught exception in looking up space point ptr... 0x803bb740, 0x2ae4f430
Caught exception in looking up space point ptr... 0x803bb8e0, 0x803bb740
Caught exception in looking up space point ptr... 0x40ffa820, 0x40ffa6d0
Caught exception in looking up space point ptr... 0x40ffa6d0, 0x40ffa820
Caught exception in looking up space point ptr... 0x40ee62e0, 0x40ee6470
Caught exception in looking up space point ptr... 0x40ee6470, 0x40ee6600
Caught exception in looking up space point ptr... 0xc5486a0, 0x669b8bd0
Caught exception in looking up space point ptr... 0x669b8bd0, 0x36449da0
Caught exception in looking up space point ptr... 0x1637f230, 0x1637ece0
Caught exception in looking up space point ptr... 0x1637ece0, 0x1637eec0
Caught exception in looking up space point ptr... 0x5fabc000, 0x589791f0
Caught exception in looking up space point ptr... 0x589791f0, 0x803b99f0
Caught exception in looking up space point ptr... 0x12523190, 0x12523350
Caught exception in looking up space point ptr... 0x12523350, 0x125230b0
Caught exception in looking up space point ptr... 0x38a973d0, 0x38a98830
Caught exception in looking up space point ptr... 0x38a98830, 0x38a973d0
Caught exception in looking up space point ptr... 0x816881c0, 0x816880d0
Caught exception in looking up space point ptr... 0x816880d0, 0x816881c0
Caught exception in looking up space point ptr... 0x38975440, 0x6b07c080
Caught exception in looking up space point ptr... 0x5fbee9a0, 0x38975440
Caught exception in looking up space point ptr... 0x6e67a930, 0x7b48a6e0
Caught exception in looking up space point ptr... 0x7b48a6e0, 0x6e67a930
Caught exception in looking up space point ptr... 0x4112c170, 0x4112bba0
Caught exception in looking up space point ptr... 0x4112bba0, 0x4112c170
Caught exception in looking up space point ptr... 0x3b34e690, 0x43590e00
Caught exception in looking up space point ptr... 0x43590e00, 0x3b34e690
Caught exception in looking up space point ptr... 0x3c618700, 0x3c464eb0
Caught exception in looking up space point ptr... 0x3c464eb0, 0x5fc5b8b0
Caught exception in looking up space point ptr... 0x33f57530, 0x33f54180
Caught exception in looking up space point ptr... 0x33f54180, 0x80050930
Caught exception in looking up space point ptr... 0x3a231730, 0x39c7f020
Caught exception in looking up space point ptr... 0x39c7f020, 0x357b4050
Caught exception in looking up space point ptr... 0x816870a0, 0x81686710
Caught exception in looking up space point ptr... 0x81686710, 0x816870a0
Caught exception in looking up space point ptr... 0x497df8b0, 0x37ab3880
Caught exception in looking up space point ptr... 0x37ab3880, 0x497df9a0
Caught exception in looking up space point ptr... 0x497df9a0, 0x497dfcc0
Caught exception in looking up space point ptr... 0x497dfcc0, 0x497df8b0
Caught exception in looking up space point ptr... 0x315f22b0, 0x315f2180
Caught exception in looking up space point ptr... 0x315f2180, 0x315f22b0
Caught exception in looking up space point ptr... 0x366e1370, 0x366dff90
Caught exception in looking up space point ptr... 0x366dff90, 0x366e1370
Caught exception in looking up space point ptr... 0x3a08c3f0, 0x3a08c7e0
Caught exception in looking up space point ptr... 0x3a08c7e0, 0x3a08c690
Caught exception in looking up space point ptr... 0x453c6320, 0xedcac30
Caught exception in looking up space point ptr... 0xedcac30, 0x453c6320
Caught exception in looking up space point ptr... 0x7ca4c090, 0x7ca4c200
Caught exception in looking up space point ptr... 0x7ca4c200, 0x7ca4cc20
Caught exception in looking up space point ptr... 0x5886f9d0, 0x58870c90
Caught exception in looking up space point ptr... 0x58870c90, 0x5886f9d0
++++>>>> total num hits: 61209, num free: 58728
04-Feb-2026 01:08:08 CET  Closed output file "np02vd_raw_run042433_0335_df-s03-d0_dw_0_20260203T204926_reco_stage0_20260204T000808_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         114.609       126.109       145.469       124.462       8.11967        19     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                        4.939e-05     9.026e-05    0.000294914   7.6811e-05    5.33364e-05      19     
produce:tpcrawdecoder:PDVDTPCReader                5.11834       6.11744       9.11319       6.00525      0.854482        19     
produce:triggerrawdecoder:PDVDTriggerReader4      0.0100793     0.0236979     0.0740502     0.0182005     0.0163431       19     
produce:timingrawdecoder:PDHDTimingRawDecoder    0.000389685   0.000778907   0.00618904    0.000463326   0.00127762       19     
produce:ctbrawdecoder:PDHDCTBRawDecoder          0.000365595   0.000468251   0.000644508   0.000446455   7.9053e-05       19     
produce:beamevent:BeamEvent                       5.17e-05     0.000138224   0.00113626    8.0761e-05    0.000236013      19     
produce:pdvddaphne:DAPHNEReaderPDVD                2.10051       2.61901       3.32715       2.46024      0.354435        19     
produce:ophit:OpHitFinder                         0.0213175     0.0337113     0.0458817     0.0334606    0.00592791       19     
produce:wclsdatavd:WireCellToolkit                 85.1104       94.9833       111.718       93.2316       6.16247        19     
produce:gaushit:GausHitFinder                     0.773933       1.21819       1.80149       1.18483      0.279733        19     
produce:nhitsfilter:NumberOfHitsFilter           0.000208792   0.000312912   0.00074549    0.000273113   0.000116452      19     
produce:reco3d:SpacePointSolver                    7.81463       14.0074       18.0294       14.2741       2.74236        19     
produce:hitpdune:DisambigFromSpacePoints          0.0700021     0.176484      0.289843      0.157221      0.0632808       19     
produce:cluster3d:Cluster3D                        0.81978       2.34028       3.70224       2.32512      0.760366        19     
[art]:TriggerResults:TriggerResultInserter       2.0611e-05    3.60132e-05   0.000154272   2.8371e-05    2.85055e-05      19     
end_path:out1:RootOutput                          3.86e-06     8.22279e-06   4.2601e-05     6.26e-06     8.23303e-06      19     
end_path:out1:RootOutput(write)                    3.87978       4.56214       6.53291       4.34757      0.569852        19     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5410.53 MB
  Peak resident set size usage (VmHWM): 3416.37 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042433_0335_df-s03-d0_dw_0_20260203T204926_reco_stage0_20260204T000808_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042433_0335_df-s03-d0_dw_0_20260203T204926_reco_stage0_20260204T000808_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042433_0335_df-s03-d0_dw_0_20260203T204926_reco_stage0_20260204T000808_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770161254.0,
    "core.end_time": 1770163689.0,
    "core.events": [
      38198,
      38204,
      38210,
      38216,
      38222,
      38228,
      38234,
      38240,
      38246,
      38252,
      38258,
      38264,
      38270,
      38276,
      38282,
      38288,
      38294,
      38300,
      38306
    ],
    "core.event_count": 19,
    "core.first_event_number": 38198,
    "core.last_event_number": 38306,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42433
    ],
    "core.runs_subruns": [
      4243300001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042433_0335_df-s03-d0_dw_0_20260203T204926.hdf5"
    }
  ]
}
justIN time: 2026-02-04 09:13:37 UTC       justIN version: 01.06.00