justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 303647.19@dunegpschedd01.fnal.gov

Jobsub ID303647.19@dunegpschedd01.fnal.gov
Workflow ID12717
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:46:24
SiteCERN
EntryCMSHTPC_T2_CH_CERN_ce506
Last heartbeat2026-02-03 23:29:12
From worker nodeHostnameb9p06p2822.cern.ch
cpuinfoAMD EPYC 7543 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit343800 (95 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 22:47:28
Input filesvd-protodune:np02vd_raw_run042421_0092_df-s03-d1_dw_0_20260203T142047.hdf5
JobscriptExit code0
Real time39m (2347s)
CPU time37m (2223s = 94%)
Max RSS bytes2609967104 (2489 MiB)
Outputting started2026-02-03 23:26:36
Output files
Finished2026-02-03 23:29:12
Saved logsjustin-logs:303647.19-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

x4c817f60
Caught exception in looking up space point ptr... 0x3b2104b0, 0x3b210690
Caught exception in looking up space point ptr... 0x5f1cc7b0, 0x5f1cc880
Caught exception in looking up space point ptr... 0x5f1cc880, 0x5f1cbf80
Caught exception in looking up space point ptr... 0x5f1cbf80, 0x3b2104b0
Caught exception in looking up space point ptr... 0x39394630, 0x30c06380
Caught exception in looking up space point ptr... 0x30c06380, 0x30c064e0
Caught exception in looking up space point ptr... 0x2ce16730, 0x3792d1f0
Caught exception in looking up space point ptr... 0x3792d1f0, 0x2ce16730
Caught exception in looking up space point ptr... 0x130a44e0, 0x3d7f6cd0
Caught exception in looking up space point ptr... 0x3d7f6cd0, 0x3d7f6bd0
Caught exception in looking up space point ptr... 0x5072a580, 0x3b2d0ff0
Caught exception in looking up space point ptr... 0x3b2d0ff0, 0x379e9ec0
Caught exception in looking up space point ptr... 0x9c90760, 0x37939b10
Caught exception in looking up space point ptr... 0x37939b10, 0x5fc7ffb0
Caught exception in looking up space point ptr... 0x5fc7ffb0, 0x45114550
Caught exception in looking up space point ptr... 0x45114550, 0x9c909c0
Caught exception in looking up space point ptr... 0x2d426a10, 0x13959e40
Caught exception in looking up space point ptr... 0x13959e40, 0x9cbec20
Caught exception in looking up space point ptr... 0x2d427b40, 0x2d426a10
Caught exception in looking up space point ptr... 0x5d500990, 0x4a4b0ce0
Caught exception in looking up space point ptr... 0x4a4b0ce0, 0x5b943e20
Caught exception in looking up space point ptr... 0x5c003690, 0x31172aa0
Caught exception in looking up space point ptr... 0x31172aa0, 0x14c218f0
Caught exception in looking up space point ptr... 0xedad940, 0xedacf40
Caught exception in looking up space point ptr... 0xedacf40, 0x4116a8b0
Caught exception in looking up space point ptr... 0xee16c50, 0x387cb410
Caught exception in looking up space point ptr... 0x387cb410, 0xee16c50
Caught exception in looking up space point ptr... 0x56f52c60, 0x10caf720
Caught exception in looking up space point ptr... 0x2c2b3100, 0x56f52c60
Caught exception in looking up space point ptr... 0x150f4400, 0x2f4de0a0
Caught exception in looking up space point ptr... 0x2f4de0a0, 0x150f4400
Caught exception in looking up space point ptr... 0x14934300, 0x4cb4c600
Caught exception in looking up space point ptr... 0x4cb4c600, 0xe5f3820
Caught exception in looking up space point ptr... 0xe5f3820, 0x58c34490
Caught exception in looking up space point ptr... 0x3131cbb0, 0x14934300
Caught exception in looking up space point ptr... 0x2b0aacf0, 0xdc00b40
Caught exception in looking up space point ptr... 0xdc00b40, 0x2b0aacf0
Caught exception in looking up space point ptr... 0x50f15e90, 0x26203060
Caught exception in looking up space point ptr... 0x5ac64d10, 0x50f15e90
Caught exception in looking up space point ptr... 0x5bd05cf0, 0x5faa0400
Caught exception in looking up space point ptr... 0x5faa0400, 0x5bd05cf0
Caught exception in looking up space point ptr... 0x362e7ea0, 0x4744ab90
Caught exception in looking up space point ptr... 0x4744ab90, 0x362e7ea0
Caught exception in looking up space point ptr... 0x2622a6e0, 0x2e5e6990
Caught exception in looking up space point ptr... 0x2e5e6990, 0x46cf2850
Caught exception in looking up space point ptr... 0x387929e0, 0x30fe9680
Caught exception in looking up space point ptr... 0x30fe9680, 0x42e95430
Caught exception in looking up space point ptr... 0x2691d3a0, 0x1395ae10
Caught exception in looking up space point ptr... 0x1395ae10, 0x2691d460
Caught exception in looking up space point ptr... 0x12ce2660, 0x4c4492c0
Caught exception in looking up space point ptr... 0x4c4492c0, 0x12ce2660
Caught exception in looking up space point ptr... 0x12ce2660, 0xede9d20
Caught exception in looking up space point ptr... 0xede9d20, 0x12ce2660
Caught exception in looking up space point ptr... 0x37a15230, 0xe73cf90
Caught exception in looking up space point ptr... 0xe73cf90, 0x37a15230
Caught exception in looking up space point ptr... 0x37a298b0, 0x3ec35ed0
Caught exception in looking up space point ptr... 0x3ec35ed0, 0x37a298b0
Caught exception in looking up space point ptr... 0x2b1ebb40, 0x2a756f30
Caught exception in looking up space point ptr... 0x2a756f30, 0x2b1ebb40
Caught exception in looking up space point ptr... 0x43807dc0, 0x43737570
Caught exception in looking up space point ptr... 0x43737570, 0x3f4052f0
Caught exception in looking up space point ptr... 0x4fa45e20, 0x311acb30
Caught exception in looking up space point ptr... 0x311acb30, 0x4fa45e20
Caught exception in looking up space point ptr... 0x36fe0dd0, 0x58002840
Caught exception in looking up space point ptr... 0x58002840, 0x36fe0dd0
++++>>>> total num hits: 26779, num free: 25657
04-Feb-2026 00:26:30 CET  Closed output file "np02vd_raw_run042421_0092_df-s03-d1_dw_0_20260203T142047_reco_stage0_20260203T232630_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         71.9525       84.9475        99.81        85.2876       6.29034        27     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                        4.965e-05    0.000111432   0.000305052   9.1201e-05    5.29145e-05      27     
produce:tpcrawdecoder:PDVDTPCReader                4.53146       4.99225       8.21607       4.88548      0.665138        27     
produce:triggerrawdecoder:PDVDTriggerReader4      0.0107644     0.0193534     0.0669333     0.0132515     0.0138178       27     
produce:timingrawdecoder:PDHDTimingRawDecoder    0.000419661    0.0010469    0.00942506    0.000513373   0.00175706       27     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.0015421    0.00214207    0.00352135    0.00199309    0.000517065      27     
produce:beamevent:BeamEvent                       5.975e-05    0.000135512   0.000997416   0.00010427    0.000170857      27     
produce:pdvddaphne:DAPHNEReaderPDVD                2.02802       2.80402        3.209        2.85542      0.301077        27     
produce:ophit:OpHitFinder                         0.0196045     0.0365715     0.0490101     0.0371304    0.00632762       27     
produce:wclsdatavd:WireCellToolkit                 54.1677       60.1261       70.6993       59.3096       4.35739        27     
produce:gaushit:GausHitFinder                     0.520623       0.90375       1.3288       0.872629      0.218968        27     
produce:nhitsfilter:NumberOfHitsFilter           0.00016007    0.00025522    0.000455242   0.000238492   5.88178e-05      27     
produce:reco3d:SpacePointSolver                    5.11032       10.6075       15.1699       10.2998       2.88163        27     
produce:hitpdune:DisambigFromSpacePoints          0.0462368     0.0947548     0.166361      0.0950765     0.0311716       27     
produce:cluster3d:Cluster3D                       0.584582       1.82872       4.54054       1.76695      0.768581        27     
[art]:TriggerResults:TriggerResultInserter       2.2581e-05    3.17175e-05    9.34e-05      2.862e-05    1.29802e-05      27     
end_path:out1:RootOutput                          4.21e-06     7.15567e-06    2.377e-05     6.17e-06     3.57508e-06      27     
end_path:out1:RootOutput(write)                    2.79459       3.50596       4.75053       3.41738      0.477036        27     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4610.27 MB
  Peak resident set size usage (VmHWM): 2609.97 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042421_0092_df-s03-d1_dw_0_20260203T142047_reco_stage0_20260203T232630_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042421_0092_df-s03-d1_dw_0_20260203T142047_reco_stage0_20260203T232630_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042421_0092_df-s03-d1_dw_0_20260203T142047_reco_stage0_20260203T232630_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770158859.0,
    "core.end_time": 1770161191.0,
    "core.events": [
      14967,
      14973,
      14979,
      14985,
      14991,
      14997,
      15003,
      15009,
      15015,
      15021,
      15027,
      15033,
      15039,
      15045,
      15051,
      15057,
      15063,
      15069,
      15075,
      15081,
      15087,
      15093,
      15099,
      15105,
      15111,
      15117,
      15123
    ],
    "core.event_count": 27,
    "core.first_event_number": 14967,
    "core.last_event_number": 15123,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42421
    ],
    "core.runs_subruns": [
      4242100001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042421_0092_df-s03-d1_dw_0_20260203T142047.hdf5"
    }
  ]
}
justIN time: 2026-02-04 05:58:09 UTC       justIN version: 01.06.00