justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 297921.78@dunegpschedd02.fnal.gov

Jobsub ID297921.78@dunegpschedd02.fnal.gov
Workflow ID12719
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:58:26
SiteUK_Oxford
EntryDUNE_UK_SGrid_Oxford_arc01
Last heartbeat2026-02-04 01:02:08
From worker nodeHostnamet2wn070.physics.ox.ac.uk
cpuinfoAMD EPYC 7763 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 23:53:17
Input filesvd-protodune:np02vd_raw_run042433_0386_df-s04-d0_dw_0_20260203T210850.hdf5
JobscriptExit code0
Real time1h (3948s)
CPU time58m (3525s = 89%)
Max RSS bytes3308331008 (3155 MiB)
Outputting started2026-02-04 00:59:07
Output files
Finished2026-02-04 01:02:08
Saved logsjustin-logs:297921.78-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

oint ptr... 0x4bea3920, 0x4bea47a0
Caught exception in looking up space point ptr... 0x4bea47a0, 0x4bea3920
Caught exception in looking up space point ptr... 0x4bc8acb0, 0x4bc8b580
Caught exception in looking up space point ptr... 0x4bc8b580, 0x4bc8acb0
Caught exception in looking up space point ptr... 0x38993680, 0x38993140
Caught exception in looking up space point ptr... 0x38993140, 0x38993680
Caught exception in looking up space point ptr... 0x4995e200, 0x4995e4a0
Caught exception in looking up space point ptr... 0x4995e4a0, 0x4995e350
Caught exception in looking up space point ptr... 0x401876f0, 0x401874f0
Caught exception in looking up space point ptr... 0x401874f0, 0x401876f0
Caught exception in looking up space point ptr... 0x6441ee00, 0x6441a010
Caught exception in looking up space point ptr... 0x6441f120, 0x6441ee00
Caught exception in looking up space point ptr... 0x3b1bdcd0, 0x3b1be1f0
Caught exception in looking up space point ptr... 0x3b1be1f0, 0x3b1bdcd0
Caught exception in looking up space point ptr... 0x454f4f50, 0x454f4500
Caught exception in looking up space point ptr... 0x454f4500, 0x454f3c00
Caught exception in looking up space point ptr... 0x4beb84c0, 0x4beb8650
Caught exception in looking up space point ptr... 0x4beb8650, 0x4beb84c0
Caught exception in looking up space point ptr... 0x6442adb0, 0x6442a800
Caught exception in looking up space point ptr... 0x6442a800, 0x6442aed0
Caught exception in looking up space point ptr... 0x68486c80, 0x8ff4050
Caught exception in looking up space point ptr... 0x8ff4050, 0x68486c80
Caught exception in looking up space point ptr... 0x68486c80, 0x8ffb9d0
Caught exception in looking up space point ptr... 0x8ffb9d0, 0x68486c80
Caught exception in looking up space point ptr... 0x68486c80, 0x68484260
Caught exception in looking up space point ptr... 0x68484260, 0x68486c80
Caught exception in looking up space point ptr... 0x18083240, 0x18082f20
Caught exception in looking up space point ptr... 0x18082f20, 0x18083240
Caught exception in looking up space point ptr... 0x5c172680, 0x33443fc0
Caught exception in looking up space point ptr... 0x33443fc0, 0x334425a0
Caught exception in looking up space point ptr... 0x187f6530, 0x187f7320
Caught exception in looking up space point ptr... 0x187f7320, 0x187f6530
Caught exception in looking up space point ptr... 0x6183afd0, 0x6183b0d0
Caught exception in looking up space point ptr... 0x6183b0d0, 0x6183afd0
Caught exception in looking up space point ptr... 0x455618f0, 0x45560350
Caught exception in looking up space point ptr... 0x45560350, 0x45560160
Caught exception in looking up space point ptr... 0xfa895b0, 0xfa883a0
Caught exception in looking up space point ptr... 0xfa883a0, 0xfa7aa60
Caught exception in looking up space point ptr... 0x187f2e50, 0x187f2fc0
Caught exception in looking up space point ptr... 0x187f2fc0, 0x187f32e0
Caught exception in looking up space point ptr... 0x2cccfa50, 0x2ccd59a0
Caught exception in looking up space point ptr... 0x2ccd59a0, 0x2ccd6650
Caught exception in looking up space point ptr... 0x5c0ca080, 0x2d33abf0
Caught exception in looking up space point ptr... 0x2d33abf0, 0x2d33ad40
Caught exception in looking up space point ptr... 0x188068d0, 0x18804bd0
Caught exception in looking up space point ptr... 0x18804bd0, 0x188049f0
Caught exception in looking up space point ptr... 0x187f7df0, 0x187f7f80
Caught exception in looking up space point ptr... 0x187f7f80, 0x187f7df0
Caught exception in looking up space point ptr... 0x3337a510, 0x3337dc40
Caught exception in looking up space point ptr... 0x3337dc40, 0x3337a510
Caught exception in looking up space point ptr... 0xe30dc60, 0xe30d7b0
Caught exception in looking up space point ptr... 0xe30d7b0, 0xe30d940
Caught exception in looking up space point ptr... 0x4240f580, 0x423eec10
Caught exception in looking up space point ptr... 0x423eec10, 0x4240f580
Caught exception in looking up space point ptr... 0x187dd160, 0x187dfa70
Caught exception in looking up space point ptr... 0x187dfa70, 0x187dced0
Caught exception in looking up space point ptr... 0x199b0d80, 0x199b0f10
Caught exception in looking up space point ptr... 0x199b0f10, 0x199b0bf0
Caught exception in looking up space point ptr... 0x199b0bf0, 0x199b0910
Caught exception in looking up space point ptr... 0x199b0910, 0x199b0a60
Caught exception in looking up space point ptr... 0x199b0a60, 0x199b07c0
Caught exception in looking up space point ptr... 0x38135110, 0x38134f80
Caught exception in looking up space point ptr... 0x38134f80, 0x381352a0
Caught exception in looking up space point ptr... 0x6df853c0, 0x6df86470
Caught exception in looking up space point ptr... 0x6df86470, 0x6df76490
Caught exception in looking up space point ptr... 0x199afd40, 0x199b0130
Caught exception in looking up space point ptr... 0x199b0130, 0x199afd40
++++>>>> total num hits: 112468, num free: 107417
04-Feb-2026 00:58:42 GMT  Closed output file "np02vd_raw_run042433_0386_df-s04-d0_dw_0_20260203T210850_reco_stage0_20260204T005842_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         169.538       201.658       224.063       201.23        14.9075        19     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       0.000109024   0.000197036   0.000356208   0.000227225   7.57796e-05      19     
produce:tpcrawdecoder:PDVDTPCReader                15.8421       20.238        26.2178       19.8712       3.21128        19     
produce:triggerrawdecoder:PDVDTriggerReader4      0.584317      0.598765      0.636429      0.592666      0.0153009       19     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.0195881     0.0199809     0.0216912     0.0198411    0.000453281      19     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.0194183     0.0200394     0.0214846     0.0199453    0.000435397      19     
produce:beamevent:BeamEvent                      0.00014392    0.000287065   0.00149815    0.000208181   0.000293617      19     
produce:pdvddaphne:DAPHNEReaderPDVD                3.55745       4.21917       5.20218       4.18483      0.342038        19     
produce:ophit:OpHitFinder                         0.028525      0.0481897     0.0584934     0.0497558    0.00687502       19     
produce:wclsdatavd:WireCellToolkit                 119.245       144.163       159.456       149.351       11.5581        19     
produce:gaushit:GausHitFinder                      1.10534       1.79834       2.63753       1.76119       0.4031         19     
produce:nhitsfilter:NumberOfHitsFilter           0.000323706   0.000489554   0.000815959   0.00045883    0.000124971      19     
produce:reco3d:SpacePointSolver                    14.1565       19.0671       24.6562       17.9313       3.65867        19     
produce:hitpdune:DisambigFromSpacePoints          0.112991      0.271046       0.43327      0.276754      0.0792401       19     
produce:cluster3d:Cluster3D                        1.74609       4.18944       16.6791       3.39126       3.13243        19     
[art]:TriggerResults:TriggerResultInserter       2.8654e-05    4.91578e-05   0.000132378   4.3682e-05    2.14929e-05      19     
end_path:out1:RootOutput                         1.0489e-05    2.74308e-05   0.000253866   1.2603e-05    5.39315e-05      19     
end_path:out1:RootOutput(write)                    5.78095       6.95185       8.15156       7.06438      0.686303        19     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5352.99 MB
  Peak resident set size usage (VmHWM): 3308.33 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042433_0386_df-s04-d0_dw_0_20260203T210850_reco_stage0_20260204T005842_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042433_0386_df-s04-d0_dw_0_20260203T210850_reco_stage0_20260204T005842_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042433_0386_df-s04-d0_dw_0_20260203T210850_reco_stage0_20260204T005842_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770162814.0,
    "core.end_time": 1770166724.0,
    "core.events": [
      44014,
      44020,
      44026,
      44032,
      44038,
      44044,
      44050,
      44056,
      44062,
      44068,
      44074,
      44080,
      44086,
      44092,
      44098,
      44104,
      44110,
      44116,
      44122
    ],
    "core.event_count": 19,
    "core.first_event_number": 44014,
    "core.last_event_number": 44122,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42433
    ],
    "core.runs_subruns": [
      4243300001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042433_0386_df-s04-d0_dw_0_20260203T210850.hdf5"
    }
  ]
}
justIN time: 2026-02-04 04:28:23 UTC       justIN version: 01.06.00