justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 303656.46@dunegpschedd01.fnal.gov

Jobsub ID303656.46@dunegpschedd01.fnal.gov
Workflow ID12717
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:56:25
SiteCERN
EntryCMSHTPC_T2_CH_CERN_ce515
Last heartbeat2026-02-04 01:15:31
From worker nodeHostnameb9p12p0201.cern.ch
cpuinfoAMD EPYC 7543 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit343800 (95 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-04 00:11:57
Input filesvd-protodune:np02vd_raw_run042421_0060_df-s02-d1_dw_0_20260203T140329.hdf5
JobscriptExit code0
Real time1h (3628s)
CPU time58m (3526s = 97%)
Max RSS bytes8074407936 (7700 MiB)
Outputting started2026-02-04 01:12:29
Output files
Finished2026-02-04 01:15:31
Saved logsjustin-logs:303656.46-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

ptr... 0x335b7c40, 0xd199e950
Caught exception in looking up space point ptr... 0xd199e950, 0x27093da0
Caught exception in looking up space point ptr... 0xa44da480, 0x3ab96c60
Caught exception in looking up space point ptr... 0x3ab96c60, 0xa44da480
Caught exception in looking up space point ptr... 0xa44da480, 0x3ab96fd0
Caught exception in looking up space point ptr... 0x3ab96fd0, 0xa44da480
Caught exception in looking up space point ptr... 0x25ee1e70, 0x25e1ffa0
Caught exception in looking up space point ptr... 0x25e1ffa0, 0x25ee1e70
Caught exception in looking up space point ptr... 0x3ce4e530, 0x2ce30dc0
Caught exception in looking up space point ptr... 0x2ce51d70, 0x3ce4e530
Caught exception in looking up space point ptr... 0x343323f0, 0x659c12b0
Caught exception in looking up space point ptr... 0x659c12b0, 0x343323f0
Caught exception in looking up space point ptr... 0x39294250, 0x44e6ccb0
Caught exception in looking up space point ptr... 0x44e6ccb0, 0x39294250
Caught exception in looking up space point ptr... 0x375f2860, 0x375f29f0
Caught exception in looking up space point ptr... 0x375f29f0, 0x8027740
Caught exception in looking up space point ptr... 0x60c1d030, 0x48c6eb40
Caught exception in looking up space point ptr... 0x48c6eb40, 0x60c1d030
Caught exception in looking up space point ptr... 0xafeb7a50, 0xafeb76c0
Caught exception in looking up space point ptr... 0x31887f00, 0xafeb7a50
Caught exception in looking up space point ptr... 0x7d926390, 0x4b6f5450
Caught exception in looking up space point ptr... 0x4b6f5450, 0xd17cfcc0
Caught exception in looking up space point ptr... 0x299c54c0, 0x4b6f57c0
Caught exception in looking up space point ptr... 0x4b6f57c0, 0x4b6f5270
Caught exception in looking up space point ptr... 0x307b5110, 0x307c5790
Caught exception in looking up space point ptr... 0x307c5790, 0x307c5920
Caught exception in looking up space point ptr... 0x2cf3c250, 0x2cf3c510
Caught exception in looking up space point ptr... 0x39c58360, 0x2cf3c250
Caught exception in looking up space point ptr... 0x2a59dd40, 0x2a5aaef0
Caught exception in looking up space point ptr... 0x36df9410, 0x2a59dd40
Caught exception in looking up space point ptr... 0x87246a0, 0x8711dc0
Caught exception in looking up space point ptr... 0x386a4420, 0x87246a0
Caught exception in looking up space point ptr... 0x34f48010, 0x3ac22430
Caught exception in looking up space point ptr... 0x34f4a720, 0x34f48010
Caught exception in looking up space point ptr... 0xa6988a70, 0x2b76bbe0
Caught exception in looking up space point ptr... 0xa6988c00, 0xa6988a70
Caught exception in looking up space point ptr... 0x48ef44f0, 0x48f00140
Caught exception in looking up space point ptr... 0x65aed5f0, 0x48ef44f0
Caught exception in looking up space point ptr... 0x516849c0, 0x45b41270
Caught exception in looking up space point ptr... 0x516850e0, 0x516849c0
Caught exception in looking up space point ptr... 0x3758e7a0, 0x3757ccd0
Caught exception in looking up space point ptr... 0x3759bf40, 0x3758e7a0
Caught exception in looking up space point ptr... 0xf4722490, 0xe2fe100
Caught exception in looking up space point ptr... 0xe2fe100, 0xe2ec310
Caught exception in looking up space point ptr... 0x2e30c470, 0x2e30c220
Caught exception in looking up space point ptr... 0x2e30c220, 0x2e30c470
Caught exception in looking up space point ptr... 0x421a6ed0, 0x53907500
Caught exception in looking up space point ptr... 0x53907500, 0x421a6ed0
Caught exception in looking up space point ptr... 0x421a6ed0, 0x421a6d90
Caught exception in looking up space point ptr... 0x421a6d90, 0x421a6ed0
Caught exception in looking up space point ptr... 0x12cf72f0, 0x1117a940
Caught exception in looking up space point ptr... 0x1117a940, 0x12cf72f0
Caught exception in looking up space point ptr... 0x60270fa0, 0x38f95f80
Caught exception in looking up space point ptr... 0x38f95f80, 0x14f81a20
Caught exception in looking up space point ptr... 0x5ff70070, 0x455a7800
Caught exception in looking up space point ptr... 0x455a7800, 0x455a7c90
Caught exception in looking up space point ptr... 0x60272520, 0x36e2f300
Caught exception in looking up space point ptr... 0x36e2f300, 0x805ce30
Caught exception in looking up space point ptr... 0x805ce30, 0x51651e00
Caught exception in looking up space point ptr... 0x51651e00, 0x4b267890
Caught exception in looking up space point ptr... 0x4e2714d0, 0x4e2716b0
Caught exception in looking up space point ptr... 0x4e2716b0, 0x4e271840
Caught exception in looking up space point ptr... 0x60a497e0, 0x4f6d6ce0
Caught exception in looking up space point ptr... 0x4f6d6ce0, 0x60a497e0
Caught exception in looking up space point ptr... 0x48c6ff30, 0x48c6e5d0
Caught exception in looking up space point ptr... 0x48c6e5d0, 0x42940040
++++>>>> total num hits: 284022, num free: 277677
04-Feb-2026 02:12:18 CET  Closed output file "np02vd_raw_run042421_0060_df-s02-d1_dw_0_20260203T140329_reco_stage0_20260204T011218_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         66.1299       132.518       454.333       97.2966       101.661        27     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                        4.418e-05    7.82257e-05   0.000304271   6.2041e-05    5.1888e-05       27     
produce:tpcrawdecoder:PDVDTPCReader                2.94392       3.33252       3.85522       3.34153      0.211264        27     
produce:triggerrawdecoder:PDVDTriggerReader4      0.0093096     0.0152033     0.0359242     0.0119097    0.00714885       27     
produce:timingrawdecoder:PDHDTimingRawDecoder    0.000365011   0.000447911   0.000727343   0.000428182   7.43244e-05      27     
produce:ctbrawdecoder:PDHDCTBRawDecoder          0.00140186    0.00169726    0.00225697    0.00163724    0.000200459      27     
produce:beamevent:BeamEvent                       4.775e-05    0.000113214   0.000800093    8.638e-05    0.000136799      27     
produce:pdvddaphne:DAPHNEReaderPDVD                1.87673       2.29901       2.70846       2.33922      0.219724        27     
produce:ophit:OpHitFinder                         0.0215879     0.0303212     0.0368741     0.0303079    0.00395452       27     
produce:wclsdatavd:WireCellToolkit                 51.5162       54.4847       59.0076       54.4272       1.78217        27     
produce:gaushit:GausHitFinder                     0.394656       1.52782       6.47474       1.07118       1.32985        27     
produce:nhitsfilter:NumberOfHitsFilter           0.00018401    0.000355984   0.000887543   0.000265781   0.000208422      27     
produce:reco3d:SpacePointSolver                    3.82564       12.9282       51.702        8.20605       13.3344        27     
produce:hitpdune:DisambigFromSpacePoints          0.0336053     0.562031       3.33807      0.150611      0.955849        27     
produce:cluster3d:Cluster3D                       0.485241       52.8927       324.979       25.6939       83.7211        27     
[art]:TriggerResults:TriggerResultInserter        1.968e-05    3.38939e-05   0.000117671    3.103e-05    1.71402e-05      27     
end_path:out1:RootOutput                          3.78e-06     2.20609e-05   0.000376682    7.64e-06     6.97634e-05      27     
end_path:out1:RootOutput(write)                    2.59533       4.4188        11.0136       3.64138       2.24532        27     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 10158.1 MB
  Peak resident set size usage (VmHWM): 8074.41 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042421_0060_df-s02-d1_dw_0_20260203T140329_reco_stage0_20260204T011218_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042421_0060_df-s02-d1_dw_0_20260203T140329_reco_stage0_20260204T011218_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042421_0060_df-s02-d1_dw_0_20260203T140329_reco_stage0_20260204T011218_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770163927.0,
    "core.end_time": 1770167539.0,
    "core.events": [
      9775,
      9781,
      9787,
      9793,
      9799,
      9805,
      9811,
      9817,
      9823,
      9829,
      9835,
      9841,
      9847,
      9853,
      9859,
      9865,
      9871,
      9877,
      9883,
      9889,
      9895,
      9901,
      9907,
      9913,
      9919,
      9925,
      9931
    ],
    "core.event_count": 27,
    "core.first_event_number": 9775,
    "core.last_event_number": 9931,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42421
    ],
    "core.runs_subruns": [
      4242100001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042421_0060_df-s02-d1_dw_0_20260203T140329.hdf5"
    }
  ]
}
justIN time: 2026-02-04 07:42:35 UTC       justIN version: 01.06.00