justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 297921.164@dunegpschedd02.fnal.gov

Jobsub ID297921.164@dunegpschedd02.fnal.gov
Workflow ID12719
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-03 22:58:26
SiteUK_QMUL
EntryDUNE_UK_London_QMUL_arcce02
Last heartbeat2026-02-04 02:10:02
From worker nodeHostnamecn046.htc.esc.qmul
cpuinfoIntel(R) Xeon(R) Gold 6248R CPU @ 3.00GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit171000 (47 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-03 23:56:34
Input filesvd-protodune:np02vd_raw_run042433_0347_df-s04-d0_dw_0_20260203T205400.hdf5
JobscriptExit code0
Real time2h (7828s)
CPU time1h (4289s = 54%)
Max RSS bytes3497095168 (3335 MiB)
Outputting started2026-02-04 02:07:10
Output files
Finished2026-02-04 02:10:02
Saved logsjustin-logs:297921.164-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

point ptr... 0x16e82f40, 0x16e83090
Caught exception in looking up space point ptr... 0x39e09340, 0x62730b40
Caught exception in looking up space point ptr... 0x62730b40, 0x61b5f120
Caught exception in looking up space point ptr... 0x32a86930, 0x16c72df0
Caught exception in looking up space point ptr... 0x16c72df0, 0x32a86930
Caught exception in looking up space point ptr... 0x3a073470, 0xfacd340
Caught exception in looking up space point ptr... 0xfacd340, 0x3a073470
Caught exception in looking up space point ptr... 0x57edbdb0, 0x4ebc54a0
Caught exception in looking up space point ptr... 0x4ebc54a0, 0x57edbdb0
Caught exception in looking up space point ptr... 0x57edbdb0, 0x57ed35c0
Caught exception in looking up space point ptr... 0x57ed35c0, 0x57edbdb0
Caught exception in looking up space point ptr... 0x57edbdb0, 0x57ed1180
Caught exception in looking up space point ptr... 0x57ed1180, 0x57edbdb0
Caught exception in looking up space point ptr... 0x435f3680, 0x435ed5d0
Caught exception in looking up space point ptr... 0x435ed5d0, 0x435f37e0
Caught exception in looking up space point ptr... 0x6b1f4ac0, 0x39e6c260
Caught exception in looking up space point ptr... 0x39e6c260, 0x6b1f4ac0
Caught exception in looking up space point ptr... 0x16e60d70, 0x7526dcf0
Caught exception in looking up space point ptr... 0x7526dcf0, 0x16e60d70
Caught exception in looking up space point ptr... 0x5186fa40, 0xa0df440
Caught exception in looking up space point ptr... 0xa0df440, 0x5186fa40
Caught exception in looking up space point ptr... 0x4edc65c0, 0x4ebe7e70
Caught exception in looking up space point ptr... 0x4ebe7e70, 0x954a3f0
Caught exception in looking up space point ptr... 0x3b8330c0, 0x376da5f0
Caught exception in looking up space point ptr... 0x376da5f0, 0x3b8330c0
Caught exception in looking up space point ptr... 0x1669ac30, 0x1669a910
Caught exception in looking up space point ptr... 0x1669a910, 0x1669ac30
Caught exception in looking up space point ptr... 0x142cd490, 0x142d0ed0
Caught exception in looking up space point ptr... 0x142d0ed0, 0x142d0ad0
Caught exception in looking up space point ptr... 0x4ebaf040, 0x4ebae2d0
Caught exception in looking up space point ptr... 0x4ebae2d0, 0x13322cc0
Caught exception in looking up space point ptr... 0x168de1e0, 0x1680e800
Caught exception in looking up space point ptr... 0x1680e800, 0x168de1e0
Caught exception in looking up space point ptr... 0x134e2a10, 0x12104190
Caught exception in looking up space point ptr... 0x12104190, 0x134e2a10
Caught exception in looking up space point ptr... 0x9549f60, 0x95483b0
Caught exception in looking up space point ptr... 0x95483b0, 0x9549f60
Caught exception in looking up space point ptr... 0x57118bb0, 0x104d4ec0
Caught exception in looking up space point ptr... 0x104d4ec0, 0x57118bb0
Caught exception in looking up space point ptr... 0x15c08040, 0x15c090d0
Caught exception in looking up space point ptr... 0x15c090d0, 0x15c08040
Caught exception in looking up space point ptr... 0x1a57dfd0, 0x739ad180
Caught exception in looking up space point ptr... 0x739ad180, 0x3c17fbe0
Caught exception in looking up space point ptr... 0x436234c0, 0xe585780
Caught exception in looking up space point ptr... 0xe585780, 0x436234c0
Caught exception in looking up space point ptr... 0x481175f0, 0x2fa20940
Caught exception in looking up space point ptr... 0x2fa20940, 0x481175f0
Caught exception in looking up space point ptr... 0x66d66c70, 0x66d66ae0
Caught exception in looking up space point ptr... 0x66d66ae0, 0x66d66c70
Caught exception in looking up space point ptr... 0x158bbf30, 0x158bbe40
Caught exception in looking up space point ptr... 0x158bbe40, 0x158bbf30
Caught exception in looking up space point ptr... 0x4edbafc0, 0x6bc58ac0
Caught exception in looking up space point ptr... 0x6bc58ac0, 0x4edbafc0
Caught exception in looking up space point ptr... 0x4edbafc0, 0x4ebaf220
Caught exception in looking up space point ptr... 0x4ebaf220, 0x4edbafc0
Caught exception in looking up space point ptr... 0x4edbafc0, 0x4edc49b0
Caught exception in looking up space point ptr... 0x4edc49b0, 0x4edbafc0
Caught exception in looking up space point ptr... 0x43694dc0, 0x3bce3210
Caught exception in looking up space point ptr... 0x3bce3210, 0x4be55720
Caught exception in looking up space point ptr... 0x380aba20, 0x63fa8850
Caught exception in looking up space point ptr... 0x63fa8850, 0x3fb028d0
Caught exception in looking up space point ptr... 0x3c315ce0, 0x82505b20
Caught exception in looking up space point ptr... 0x82505b20, 0x3c315ce0
Caught exception in looking up space point ptr... 0x518b3f90, 0x4adcba50
Caught exception in looking up space point ptr... 0x4adcba50, 0x518b3f90
Caught exception in looking up space point ptr... 0x725b98c0, 0x725bce60
Caught exception in looking up space point ptr... 0x725bce60, 0x725b9d70
++++>>>> total num hits: 81817, num free: 78722
04-Feb-2026 02:06:52 GMT  Closed output file "np02vd_raw_run042433_0347_df-s04-d0_dw_0_20260203T205400_reco_stage0_20260204T020652_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         341.816       407.011       490.718       412.344       42.2554        19     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       9.3393e-05    0.000127949   0.000313587   0.000113881   4.72406e-05      19     
produce:tpcrawdecoder:PDVDTPCReader                132.486       178.74        249.567       173.271       29.2302        19     
produce:triggerrawdecoder:PDVDTriggerReader4       0.59935       0.91722       1.5933       0.823824      0.262218        19     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.0165397     0.0289995     0.0870765     0.0202755     0.0198894       19     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.0165265     0.0167897     0.0174394     0.0167222    0.000234734      19     
produce:beamevent:BeamEvent                      0.000130684   0.000233418   0.00136585    0.000169676   0.000268152      19     
produce:pdvddaphne:DAPHNEReaderPDVD                13.157        18.6246       25.8142       18.5744       2.9038         19     
produce:ophit:OpHitFinder                         0.0328045     0.0513324     0.0696838     0.0481514     0.0121921       19     
produce:wclsdatavd:WireCellToolkit                 140.144       171.86        227.258       174.27        27.0766        19     
produce:gaushit:GausHitFinder                      1.21338       1.99491       3.01347       1.89811      0.559829        19     
produce:nhitsfilter:NumberOfHitsFilter           0.000416589   0.000779412   0.00144065    0.000787204   0.000221021      19     
produce:reco3d:SpacePointSolver                    8.89995       22.8654       39.4657       19.4698       8.76665        19     
produce:hitpdune:DisambigFromSpacePoints          0.107374      0.349049       1.18485      0.255376      0.223408        19     
produce:cluster3d:Cluster3D                        1.1516        5.03014       41.3346       2.56577       8.6486         19     
[art]:TriggerResults:TriggerResultInserter       2.7346e-05    4.56662e-05   9.0936e-05    4.1412e-05    1.61563e-05      19     
end_path:out1:RootOutput                          4.663e-06    1.91327e-05   0.000210366    7.428e-06    4.52525e-05      19     
end_path:out1:RootOutput(write)                    5.1681        6.48044       8.65366       5.97687       1.27872        19     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5448.23 MB
  Peak resident set size usage (VmHWM): 3497.1 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042433_0347_df-s04-d0_dw_0_20260203T205400_reco_stage0_20260204T020652_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042433_0347_df-s04-d0_dw_0_20260203T205400_reco_stage0_20260204T020652_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042433_0347_df-s04-d0_dw_0_20260203T205400_reco_stage0_20260204T020652_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770163003.0,
    "core.end_time": 1770170816.0,
    "core.events": [
      39568,
      39574,
      39580,
      39586,
      39592,
      39598,
      39604,
      39610,
      39616,
      39622,
      39628,
      39634,
      39640,
      39646,
      39652,
      39658,
      39664,
      39670,
      39676
    ],
    "core.event_count": 19,
    "core.first_event_number": 39568,
    "core.last_event_number": 39676,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42433
    ],
    "core.runs_subruns": [
      4243300001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042433_0347_df-s04-d0_dw_0_20260203T205400.hdf5"
    }
  ]
}
justIN time: 2026-02-04 04:29:06 UTC       justIN version: 01.06.00