justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 297923.86@dunegpschedd02.fnal.gov

Jobsub ID297923.86@dunegpschedd02.fnal.gov
Workflow ID12719
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 23:00:26
SiteNL_NIKHEF
EntryVIRGO_NL_NIKHEF_dissel_multicore
Last heartbeat2026-02-04 01:25:38
From worker nodeHostnamewn-pijl-011.farm.nikhef.nl
cpuinfoAMD EPYC 9754 128-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit129600 (36 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-04 00:00:21
Input filesvd-protodune:np02vd_raw_run042433_0448_df-s03-d1_dw_0_20260203T213224.hdf5
JobscriptExit code0
Real time1h (4950s)
CPU time1h (4674s = 94%)
Max RSS bytes3635838976 (3467 MiB)
Outputting started2026-02-04 01:22:54
Output files
Finished2026-02-04 01:25:38
Saved logsjustin-logs:297923.86-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

0x7a980970, 0x9c4ba4d0
Caught exception in looking up space point ptr... 0x4ff1b6a0, 0x704dd310
Caught exception in looking up space point ptr... 0x704dd310, 0x447c0380
Caught exception in looking up space point ptr... 0x668d9280, 0x9cdd8850
Caught exception in looking up space point ptr... 0x668cc440, 0x668d9280
Caught exception in looking up space point ptr... 0x4e2a3f50, 0x780d2f40
Caught exception in looking up space point ptr... 0xa78164c0, 0x4e2a3f50
Caught exception in looking up space point ptr... 0x6cb0cfa0, 0x6cb0fd60
Caught exception in looking up space point ptr... 0x94104240, 0x6cb0cfa0
Caught exception in looking up space point ptr... 0xa77b3970, 0x51a3d700
Caught exception in looking up space point ptr... 0xa781f380, 0xa77b3970
Caught exception in looking up space point ptr... 0x8f994980, 0x805c2660
Caught exception in looking up space point ptr... 0x805c2660, 0x46f72e50
Caught exception in looking up space point ptr... 0x49eea820, 0x49edcbd0
Caught exception in looking up space point ptr... 0x49edcbd0, 0x49ed03d0
Caught exception in looking up space point ptr... 0x50385380, 0x93af4df0
Caught exception in looking up space point ptr... 0x93af4df0, 0x50386100
Caught exception in looking up space point ptr... 0x46db1bf0, 0x8226d630
Caught exception in looking up space point ptr... 0x8226d630, 0x8226d4a0
Caught exception in looking up space point ptr... 0x704b30b0, 0x704b2f20
Caught exception in looking up space point ptr... 0x704b2f20, 0x704b2750
Caught exception in looking up space point ptr... 0x7b550cf0, 0x7b567150
Caught exception in looking up space point ptr... 0x7b567150, 0x7b5672e0
Caught exception in looking up space point ptr... 0x7b5672e0, 0x444c4c50
Caught exception in looking up space point ptr... 0x444c4c50, 0x7b550cf0
Caught exception in looking up space point ptr... 0x70c54800, 0x9f5f4a10
Caught exception in looking up space point ptr... 0x9f5f4a10, 0x9cfebd70
Caught exception in looking up space point ptr... 0x49141e40, 0x4912f6f0
Caught exception in looking up space point ptr... 0x51cc4660, 0x49141e40
Caught exception in looking up space point ptr... 0x42664dc0, 0x70484e70
Caught exception in looking up space point ptr... 0x70484380, 0x42664dc0
Caught exception in looking up space point ptr... 0xab7c4170, 0x704cbb40
Caught exception in looking up space point ptr... 0x704cbb40, 0x4ff3dab0
Caught exception in looking up space point ptr... 0x9fcebd10, 0x9fceb9c0
Caught exception in looking up space point ptr... 0x9fcebea0, 0x9fcebd10
Caught exception in looking up space point ptr... 0x46512400, 0x465126a0
Caught exception in looking up space point ptr... 0x465126a0, 0xab830670
Caught exception in looking up space point ptr... 0xab8308c0, 0x46512400
Caught exception in looking up space point ptr... 0xb7f55c60, 0xb7f55a10
Caught exception in looking up space point ptr... 0xb7f55a10, 0xb7f5c6c0
Caught exception in looking up space point ptr... 0x6fdaa5f0, 0x6fdaa3a0
Caught exception in looking up space point ptr... 0x6fdaa3a0, 0x6fdabf60
Caught exception in looking up space point ptr... 0x9c9f7670, 0x9c9f7800
Caught exception in looking up space point ptr... 0x9c9f7800, 0x9c9ebe30
Caught exception in looking up space point ptr... 0x8d621590, 0xa25c3440
Caught exception in looking up space point ptr... 0xa25c3440, 0x9af73980
Caught exception in looking up space point ptr... 0x9af73980, 0x9af736e0
Caught exception in looking up space point ptr... 0x4d0bb390, 0x4d0bbd10
Caught exception in looking up space point ptr... 0x4d0bbbe0, 0x4d0bb390
Caught exception in looking up space point ptr... 0x9f609510, 0x9d040040
Caught exception in looking up space point ptr... 0x528d4c70, 0x9f609510
Caught exception in looking up space point ptr... 0x4db46f00, 0x4db38630
Caught exception in looking up space point ptr... 0x4db38630, 0x6a2348f0
Caught exception in looking up space point ptr... 0x41c0c110, 0x41c1be30
Caught exception in looking up space point ptr... 0x41bfc260, 0x41c0c110
Caught exception in looking up space point ptr... 0x9ae55480, 0x439c4900
Caught exception in looking up space point ptr... 0x439c4900, 0x439c4a90
Caught exception in looking up space point ptr... 0x93863460, 0xc5d897c0
Caught exception in looking up space point ptr... 0x93863910, 0x93863460
Caught exception in looking up space point ptr... 0x62fbfd20, 0x8b682db0
Caught exception in looking up space point ptr... 0x53747310, 0x62fbfd20
Caught exception in looking up space point ptr... 0x95433690, 0x953fe7d0
Caught exception in looking up space point ptr... 0x95433820, 0x95433690
Caught exception in looking up space point ptr... 0x9fdb43c0, 0x9fdc67f0
Caught exception in looking up space point ptr... 0x9fda1f90, 0x9fdb43c0
Caught exception in looking up space point ptr... 0x79d6ee60, 0x79d7e130
Caught exception in looking up space point ptr... 0x79d7e130, 0x79d8d400
++++>>>> total num hits: 490479, num free: 481837
04-Feb-2026 02:22:37 CET  Closed output file "np02vd_raw_run042433_0448_df-s03-d1_dw_0_20260203T213224_reco_stage0_20260204T012237_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         184.623       254.773       309.477       273.88        42.6569        19     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       8.9835e-05    0.000127959   0.000329508   0.000114754   5.08894e-05      19     
produce:tpcrawdecoder:PDVDTPCReader                16.9988       17.6481       18.446        17.6826      0.346132        19     
produce:triggerrawdecoder:PDVDTriggerReader4       0.52086      0.544554      0.638047      0.532846      0.0262225       19     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.0177004     0.0197447     0.0509259     0.0180823    0.00735241       19     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.0178343     0.018094      0.0186808     0.0180096    0.00024557       19     
produce:beamevent:BeamEvent                      0.000130918   0.000250536   0.00119823    0.000193262   0.000227117      19     
produce:pdvddaphne:DAPHNEReaderPDVD                3.56396       3.95365       4.35807       3.97422      0.252813        19     
produce:ophit:OpHitFinder                         0.0389613     0.0456502     0.0519632     0.0468446    0.00431393       19     
produce:wclsdatavd:WireCellToolkit                 129.061       150.514        165.8        151.204       9.43075        19     
produce:gaushit:GausHitFinder                      1.61252       2.75945       4.19508       2.86115      0.713876        19     
produce:nhitsfilter:NumberOfHitsFilter           0.00039043    0.00162113     0.0029382    0.00185258    0.000736282      19     
produce:reco3d:SpacePointSolver                    15.5991       21.6666       35.6202       19.9576       5.12287        19     
produce:hitpdune:DisambigFromSpacePoints          0.164373      0.373644      0.600511      0.360983      0.111618        19     
produce:cluster3d:Cluster3D                        2.23756       48.3312       89.0555       74.011        37.452         19     
[art]:TriggerResults:TriggerResultInserter       2.8563e-05    6.39255e-05   0.000169115   5.2359e-05    3.42341e-05      19     
end_path:out1:RootOutput                          4.647e-06    9.58958e-06    4.03e-05      6.259e-06    8.25504e-06      19     
end_path:out1:RootOutput(write)                    6.06965       8.79548       10.8532       9.02756       1.42478        19     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5587.79 MB
  Peak resident set size usage (VmHWM): 3635.84 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042433_0448_df-s03-d1_dw_0_20260203T213224_reco_stage0_20260204T012237_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042433_0448_df-s03-d1_dw_0_20260203T213224_reco_stage0_20260204T012237_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042433_0448_df-s03-d1_dw_0_20260203T213224_reco_stage0_20260204T012237_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770163235.0,
    "core.end_time": 1770168160.0,
    "core.events": [
      51081,
      51087,
      51093,
      51099,
      51105,
      51111,
      51117,
      51123,
      51129,
      51135,
      51141,
      51147,
      51153,
      51159,
      51165,
      51171,
      51177,
      51183,
      51189
    ],
    "core.event_count": 19,
    "core.first_event_number": 51081,
    "core.last_event_number": 51189,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42433
    ],
    "core.runs_subruns": [
      4243300001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042433_0448_df-s03-d1_dw_0_20260203T213224.hdf5"
    }
  ]
}
justIN time: 2026-02-04 06:00:26 UTC       justIN version: 01.06.00