justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 303647.44@dunegpschedd01.fnal.gov

Jobsub ID303647.44@dunegpschedd01.fnal.gov
Workflow ID12717
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:46:24
SiteCERN
EntryCMSHTPC_T2_CH_CERN_ce511
Last heartbeat2026-02-03 23:28:54
From worker nodeHostnameb9p02p3983.cern.ch
cpuinfoAMD EPYC 7543 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit343800 (95 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 22:46:49
Input filesvd-protodune:np02vd_raw_run042421_0082_df-s02-d0_dw_0_20260203T141522.hdf5
JobscriptExit code0
Real time39m (2386s)
CPU time37m (2279s = 95%)
Max RSS bytes2646560768 (2523 MiB)
Outputting started2026-02-03 23:26:37
Output files
Finished2026-02-03 23:28:54
Saved logsjustin-logs:303647.44-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

0x2ee6ba30
Caught exception in looking up space point ptr... 0x53e7c150, 0x53e7c280
Caught exception in looking up space point ptr... 0x53e7c280, 0x53e7c3b0
Caught exception in looking up space point ptr... 0x53e7c3b0, 0x53e7e530
Caught exception in looking up space point ptr... 0x4669b690, 0x4669ab10
Caught exception in looking up space point ptr... 0x4669ab10, 0x4669b690
Caught exception in looking up space point ptr... 0x4669b690, 0x4669b250
Caught exception in looking up space point ptr... 0x4669b250, 0x4669b690
Caught exception in looking up space point ptr... 0xe8bd8e0, 0xe8bc110
Caught exception in looking up space point ptr... 0xe8bc110, 0xe8bbdd0
Caught exception in looking up space point ptr... 0x3d6a02a0, 0x15926120
Caught exception in looking up space point ptr... 0x15926120, 0x3d6a0000
Caught exception in looking up space point ptr... 0x28ed5d60, 0x429e4120
Caught exception in looking up space point ptr... 0x429e4120, 0x28ed5d60
Caught exception in looking up space point ptr... 0x28ed5d60, 0xa442480
Caught exception in looking up space point ptr... 0xa442480, 0x28ed5d60
Caught exception in looking up space point ptr... 0x2ee65090, 0x2ee5d910
Caught exception in looking up space point ptr... 0x2ee5d910, 0x2ee65090
Caught exception in looking up space point ptr... 0x3dd78450, 0x3dd78300
Caught exception in looking up space point ptr... 0x3dd78300, 0x3dd78450
Caught exception in looking up space point ptr... 0x3d69f150, 0x3d69ee30
Caught exception in looking up space point ptr... 0x3d69ee30, 0x3d69f150
Caught exception in looking up space point ptr... 0x65d425a0, 0x65d3a170
Caught exception in looking up space point ptr... 0x65d406c0, 0x65d425a0
Caught exception in looking up space point ptr... 0x440c7f10, 0x3e304050
Caught exception in looking up space point ptr... 0x3e304050, 0x440c7f10
Caught exception in looking up space point ptr... 0x440c7f10, 0x3d69add0
Caught exception in looking up space point ptr... 0x3d69add0, 0x440c7f10
Caught exception in looking up space point ptr... 0x112ce680, 0x4baf7fa0
Caught exception in looking up space point ptr... 0x92fd570, 0xec0b4f0
Caught exception in looking up space point ptr... 0xec0b4f0, 0x5dc98c20
Caught exception in looking up space point ptr... 0x5dc98c20, 0x112ce680
Caught exception in looking up space point ptr... 0x28d97da0, 0x28d97a00
Caught exception in looking up space point ptr... 0x28d97a00, 0x28d97da0
Caught exception in looking up space point ptr... 0x254e1a10, 0x254e1680
Caught exception in looking up space point ptr... 0x254e1680, 0x254e1a10
Caught exception in looking up space point ptr... 0x3dd66670, 0x3dd66770
Caught exception in looking up space point ptr... 0x3dd66770, 0x3dd668c0
Caught exception in looking up space point ptr... 0x1591f7f0, 0x1591f340
Caught exception in looking up space point ptr... 0x1591f340, 0x1591f7f0
Caught exception in looking up space point ptr... 0x1591f7f0, 0x1591f020
Caught exception in looking up space point ptr... 0x1591f020, 0x1591f7f0
Caught exception in looking up space point ptr... 0x3ecf5800, 0x3ecf66f0
Caught exception in looking up space point ptr... 0x3ecf66f0, 0x3ecf5b20
Caught exception in looking up space point ptr... 0x62407fb0, 0x4a77cc80
Caught exception in looking up space point ptr... 0x4a77cc80, 0x2cfdb370
Caught exception in looking up space point ptr... 0x3178ecf0, 0x2f7a6900
Caught exception in looking up space point ptr... 0x2f7a6900, 0x94330b0
Caught exception in looking up space point ptr... 0xda27c30, 0x28f6d8d0
Caught exception in looking up space point ptr... 0x28f6d8d0, 0xda27c30
Caught exception in looking up space point ptr... 0xda27c30, 0x51765cb0
Caught exception in looking up space point ptr... 0x51765cb0, 0xda27c30
Caught exception in looking up space point ptr... 0x275f72d0, 0x11f4e230
Caught exception in looking up space point ptr... 0x11f4e230, 0x3fc8a190
Caught exception in looking up space point ptr... 0x567918d0, 0x56795a90
Caught exception in looking up space point ptr... 0x56795a90, 0x56796300
Caught exception in looking up space point ptr... 0x28d7ef20, 0x28d82c90
Caught exception in looking up space point ptr... 0x28d82c90, 0x28d82f50
Caught exception in looking up space point ptr... 0xda29360, 0x3c546790
Caught exception in looking up space point ptr... 0x3c546790, 0x14bcec10
Caught exception in looking up space point ptr... 0x21587dc0, 0x215880e0
Caught exception in looking up space point ptr... 0x216a0f70, 0x21587dc0
Caught exception in looking up space point ptr... 0x4d1622e0, 0x4d1633b0
Caught exception in looking up space point ptr... 0x4d1633b0, 0x4d1632d0
Caught exception in looking up space point ptr... 0x8972ae0, 0x2916abe0
Caught exception in looking up space point ptr... 0x2916abe0, 0xdbaf620
++++>>>> total num hits: 66491, num free: 64483
04-Feb-2026 00:26:27 CET  Closed output file "np02vd_raw_run042421_0082_df-s02-d0_dw_0_20260203T141522_reco_stage0_20260203T232627_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         70.4783       85.8009       95.3254       86.994        6.6289         27     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                        4.313e-05    8.2915e-05    0.000219191    7.123e-05    3.70442e-05      27     
produce:tpcrawdecoder:PDVDTPCReader                3.83934       4.69452       14.7244       4.28247       2.00293        27     
produce:triggerrawdecoder:PDVDTriggerReader4      0.0103044     0.0140565     0.0446726     0.0126366    0.00618815       27     
produce:timingrawdecoder:PDHDTimingRawDecoder    0.000446472   0.000539342   0.000926423   0.000508921   0.000105035      27     
produce:ctbrawdecoder:PDHDCTBRawDecoder          0.00160417    0.00197809    0.00406724    0.00179775    0.00051478       27     
produce:beamevent:BeamEvent                       5.966e-05    0.000125063   0.000962682    8.551e-05    0.000166158      27     
produce:pdvddaphne:DAPHNEReaderPDVD                2.21415       2.75018       3.27464       2.75236       0.30844        27     
produce:ophit:OpHitFinder                         0.0258515     0.0349791     0.0445286     0.035404     0.00526258       27     
produce:wclsdatavd:WireCellToolkit                 54.5427       62.7637       74.6101       62.7018       4.92892        27     
produce:gaushit:GausHitFinder                     0.467638      0.854689       1.41853      0.777726       0.2545         27     
produce:nhitsfilter:NumberOfHitsFilter           0.000161791    0.000259     0.000870093   0.000228069   0.000130213      27     
produce:reco3d:SpacePointSolver                    2.29171       9.10554       17.7735       8.08887       3.79981        27     
produce:hitpdune:DisambigFromSpacePoints          0.026409      0.0820039      0.13393      0.0731531     0.0301874       27     
produce:cluster3d:Cluster3D                       0.375197       1.6734        4.47358       1.57674      0.860865        27     
[art]:TriggerResults:TriggerResultInserter        1.778e-05    3.26756e-05    9.158e-05     2.871e-05    1.37374e-05      27     
end_path:out1:RootOutput                          4.11e-06     6.65963e-06    2.747e-05      5.5e-06     4.22207e-06      27     
end_path:out1:RootOutput(write)                    2.96384       3.80303       4.81297       3.71057       0.51549        27     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4617.24 MB
  Peak resident set size usage (VmHWM): 2646.56 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042421_0082_df-s02-d0_dw_0_20260203T141522_reco_stage0_20260203T232627_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042421_0082_df-s02-d0_dw_0_20260203T141522_reco_stage0_20260203T232627_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042421_0082_df-s02-d0_dw_0_20260203T141522_reco_stage0_20260203T232627_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770158821.0,
    "core.end_time": 1770161188.0,
    "core.events": [
      13344,
      13350,
      13356,
      13362,
      13368,
      13374,
      13380,
      13386,
      13392,
      13398,
      13404,
      13410,
      13416,
      13422,
      13428,
      13434,
      13440,
      13446,
      13452,
      13458,
      13464,
      13470,
      13476,
      13482,
      13488,
      13494,
      13500
    ],
    "core.event_count": 27,
    "core.first_event_number": 13344,
    "core.last_event_number": 13500,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42421
    ],
    "core.runs_subruns": [
      4242100001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042421_0082_df-s02-d0_dw_0_20260203T141522.hdf5"
    }
  ]
}
justIN time: 2026-02-04 07:45:09 UTC       justIN version: 01.06.00