justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 297982.0@dunegpschedd02.fnal.gov

Jobsub ID297982.0@dunegpschedd02.fnal.gov
Workflow ID12719
Stage ID1
User nameykermaid@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Submitted time2026-02-04 00:48:30
SiteUK_QMUL
EntryDUNE_UK_London_QMUL_arcce03
Last heartbeat2026-02-04 02:54:28
From worker nodeHostnamecn081.htc.esc.qmul
cpuinfoIntel(R) Xeon(R) Gold 6248R CPU @ 3.00GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit171000 (47 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-02-04 00:49:27
Input filesvd-protodune:np02vd_raw_run042433_0125_df-s04-d0_dw_0_20260203T192938.hdf5
JobscriptExit code0
Real time2h (7361s)
CPU time54m (3268s = 44%)
Max RSS bytes3334512640 (3180 MiB)
Outputting started2026-02-04 02:52:09
Output files
Finished2026-02-04 02:54:28
Saved logsjustin-logs:297982.0-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

looking up space point ptr... 0xcfca810, 0xcfcbf80
Caught exception in looking up space point ptr... 0xcfcbf80, 0x83118120
Caught exception in looking up space point ptr... 0x83118120, 0x80d73120
Caught exception in looking up space point ptr... 0x80d73120, 0xcfdc7b0
Caught exception in looking up space point ptr... 0xcfbd0f0, 0xcfbcf10
Caught exception in looking up space point ptr... 0xcfbcf10, 0x46bef7a0
Caught exception in looking up space point ptr... 0x5ecd5830, 0x5ecd5680
Caught exception in looking up space point ptr... 0x5ecd0390, 0x5ecd5830
Caught exception in looking up space point ptr... 0x819ac870, 0x819ac550
Caught exception in looking up space point ptr... 0x819ac550, 0x819ac6e0
Caught exception in looking up space point ptr... 0xd2236d0, 0xd223560
Caught exception in looking up space point ptr... 0xd223560, 0xd2236d0
Caught exception in looking up space point ptr... 0xd5c94c0, 0xd5c9650
Caught exception in looking up space point ptr... 0xd5c9650, 0xd5c97e0
Caught exception in looking up space point ptr... 0x10bde8b0, 0x10bde720
Caught exception in looking up space point ptr... 0x10bde720, 0x10bde8b0
Caught exception in looking up space point ptr... 0x61ad9a40, 0x61ad9800
Caught exception in looking up space point ptr... 0x61ad9800, 0x61ad9900
Caught exception in looking up space point ptr... 0x4c7eab70, 0x4c7eac50
Caught exception in looking up space point ptr... 0x4c7eac50, 0x4c7eab70
Caught exception in looking up space point ptr... 0x507e4910, 0x507e4610
Caught exception in looking up space point ptr... 0x507e4610, 0x507e4910
Caught exception in looking up space point ptr... 0x507b53a0, 0x507bfe50
Caught exception in looking up space point ptr... 0x507bfe50, 0x37de4a10
Caught exception in looking up space point ptr... 0x37de4a10, 0x507b51c0
Caught exception in looking up space point ptr... 0x604bc680, 0x3c56e670
Caught exception in looking up space point ptr... 0x3c56e670, 0x604bb710
Caught exception in looking up space point ptr... 0xe152e10, 0xe152b60
Caught exception in looking up space point ptr... 0xe152b60, 0xe152e10
Caught exception in looking up space point ptr... 0xd5e3a00, 0xd5e3670
Caught exception in looking up space point ptr... 0xd5e3670, 0xd5e3a00
Caught exception in looking up space point ptr... 0x310f1330, 0x310f1480
Caught exception in looking up space point ptr... 0x310f1480, 0x310f1330
Caught exception in looking up space point ptr... 0x36c580c0, 0x36c57f80
Caught exception in looking up space point ptr... 0x36c57f80, 0x36c580c0
Caught exception in looking up space point ptr... 0x80f28a20, 0x80f28890
Caught exception in looking up space point ptr... 0x80f28890, 0x80f28720
Caught exception in looking up space point ptr... 0x821435a0, 0x3ebed300
Caught exception in looking up space point ptr... 0x3ebed300, 0x5a5afb00
Caught exception in looking up space point ptr... 0x3ec51420, 0x4e491540
Caught exception in looking up space point ptr... 0x3ec51990, 0x3ec51420
Caught exception in looking up space point ptr... 0x50a96770, 0x50d6d640
Caught exception in looking up space point ptr... 0x50d6d640, 0x6028e0c0
Caught exception in looking up space point ptr... 0x80f1f070, 0x80f1f2b0
Caught exception in looking up space point ptr... 0x80f1f2b0, 0x80f1f9a0
Caught exception in looking up space point ptr... 0x37def710, 0x37df5ac0
Caught exception in looking up space point ptr... 0x37df5ac0, 0x37df3040
Caught exception in looking up space point ptr... 0x36e0a200, 0xd15f950
Caught exception in looking up space point ptr... 0xd15f950, 0x36e0a200
Caught exception in looking up space point ptr... 0x36e0a200, 0x17e0ab80
Caught exception in looking up space point ptr... 0x17e0ab80, 0x36e0a200
Caught exception in looking up space point ptr... 0x507df920, 0x507df6c0
Caught exception in looking up space point ptr... 0x507df6c0, 0x507e5910
Caught exception in looking up space point ptr... 0x149a9400, 0x604abc00
Caught exception in looking up space point ptr... 0x604abc00, 0x149a9400
Caught exception in looking up space point ptr... 0x5bdf5480, 0x5bdf8ad0
Caught exception in looking up space point ptr... 0x5bdf8ad0, 0x5bdf51e0
Caught exception in looking up space point ptr... 0x36c658d0, 0x41b011b0
Caught exception in looking up space point ptr... 0x41b011b0, 0x36c6d290
Caught exception in looking up space point ptr... 0xee22b60, 0x387e30a0
Caught exception in looking up space point ptr... 0x387e30a0, 0xee22b60
Caught exception in looking up space point ptr... 0xee22b60, 0xee33720
Caught exception in looking up space point ptr... 0xee33720, 0xee22b60
Caught exception in looking up space point ptr... 0x3c5806f0, 0x3c580490
Caught exception in looking up space point ptr... 0x3c580490, 0x7e861f90
Caught exception in looking up space point ptr... 0x507d96e0, 0x507d3580
Caught exception in looking up space point ptr... 0x507d3580, 0x507d5790
++++>>>> total num hits: 129582, num free: 124381
04-Feb-2026 02:51:58 GMT  Closed output file "np02vd_raw_run042433_0125_df-s04-d0_dw_0_20260203T192938_reco_stage0_20260204T025158_offline.root"

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         333.223       381.718       441.784       382.943       32.5591        19     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       8.8413e-05    0.000118601   0.000243841   0.000117286   3.27321e-05      19     
produce:tpcrawdecoder:PDVDTPCReader                162.907       205.281       262.785       195.923       30.5481        19     
produce:triggerrawdecoder:PDVDTriggerReader4      0.576808       1.36351       5.37148      0.974491       1.09111        19     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.0168063     0.0190223     0.0463123     0.017025     0.00661564       19     
produce:ctbrawdecoder:PDHDCTBRawDecoder           0.0166096     0.0188528     0.0503386     0.0169274    0.00743463       19     
produce:beamevent:BeamEvent                      0.000118111   0.000224541   0.00125883     0.0001659    0.000245747      19     
produce:pdvddaphne:DAPHNEReaderPDVD                12.776        19.3817       23.2554       19.4168       2.55661        19     
produce:ophit:OpHitFinder                         0.0319833     0.0420041     0.0671028     0.0367437     0.0106527       19     
produce:wclsdatavd:WireCellToolkit                 117.433       130.231       189.374       125.611       15.6309        19     
produce:gaushit:GausHitFinder                      1.01383       1.51859       2.06927       1.48715      0.275388        19     
produce:nhitsfilter:NumberOfHitsFilter           0.00050706    0.000633135   0.00108773    0.000614891   0.000121678      19     
produce:reco3d:SpacePointSolver                    11.235        16.4888       26.8216       16.4595       3.56715        19     
produce:hitpdune:DisambigFromSpacePoints          0.170208      0.233285       0.40499      0.225943      0.0549616       19     
produce:cluster3d:Cluster3D                        1.25758       2.33099       4.94633       2.20814      0.821629        19     
[art]:TriggerResults:TriggerResultInserter       2.6544e-05    3.36877e-05   0.000102223   2.9444e-05    1.66175e-05      19     
end_path:out1:RootOutput                          4.299e-06    7.50353e-06    2.749e-05     6.731e-06    4.78689e-06      19     
end_path:out1:RootOutput(write)                    4.29247       4.76446       7.29155       4.62948      0.657647        19     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5299.74 MB
  Peak resident set size usage (VmHWM): 3334.51 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run042433_0125_df-s04-d0_dw_0_20260203T192938_reco_stage0_20260204T025158_offline.root
Forming reco metadata
Successfully opened file np02vd_raw_run042433_0125_df-s04-d0_dw_0_20260203T192938_reco_stage0_20260204T025158_offline.root
Ran successfully
{
  "name": "np02vd_raw_run042433_0125_df-s04-d0_dw_0_20260203T192938_reco_stage0_20260204T025158_offline.root",
  "namespace": "usertests",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_17_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage0_protodunevd_offline.fcl",
    "dune.campaign": "vd-protodune-det-reco-test",
    "core.start_time": 1770166178.0,
    "core.end_time": 1770173523.0,
    "core.events": [
      14260,
      14266,
      14272,
      14278,
      14284,
      14290,
      14296,
      14302,
      14308,
      14314,
      14320,
      14326,
      14332,
      14338,
      14344,
      14350,
      14356,
      14362,
      14368
    ],
    "core.event_count": 19,
    "core.first_event_number": 14260,
    "core.last_event_number": 14368,
    "core.data_stream": "cosmics",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      42433
    ],
    "core.runs_subruns": [
      4243300001
    ]
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run042433_0125_df-s04-d0_dw_0_20260203T192938.hdf5"
    }
  ]
}
justIN time: 2026-02-04 04:26:58 UTC       justIN version: 01.06.00