justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 257047.23@dunegpschedd02.fnal.gov

Jobsub ID257047.23@dunegpschedd02.fnal.gov
Workflow ID10977
Stage ID1
User nameamoor@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes2097152000 (2000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-12-03 19:53:40
SiteNL_SURFsara
EntryDUNE_SurfSARA_arc03
Last heartbeat2025-12-03 20:50:00
From worker nodeHostnamewn-lb-10.gina.surf.nl
cpuinfoAMD EPYC 9754 128-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes2097152000 (2000 MiB)
Wall seconds limit129600 (36 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Started2025-12-03 19:54:43
Input filesusertests:000644_reco_data_2025-12-01T_121059Z_reco_data_2025-12-03T_160221Z_reco_data_2025-12-03T_174006Z.root
JobscriptExit code0
Real time54m (3293s)
CPU time54m (3282s = 99%)
Max RSS bytes7079739392 (6751 MiB)
Outputting started2025-12-03 20:49:38
Output files
Finished2025-12-03 20:50:00
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

AFM detsim jobscript.
Input PFN = root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/usertests/a4/f1/000644_reco_data_2025-12-01T_121059Z_reco_data_2025-12-03T_160221Z_reco_data_2025-12-03T_174006Z.root
Setting up larsoft UPS area... /cvmfs/larsoft.opensciencegrid.org
Setting up DUNE UPS area... /cvmfs/dune.opensciencegrid.org/products/dune/
/cvmfs/larsoft.opensciencegrid.org/products/xrootd/v5_5_5a/Linux64bit+3.10-2.17-e26-p3915-prof/lib/libXrdPosixPreload.so
=== Start last 50 lines of lar log file ===
[20:55:35.524] W [sigproc ] <OmnibusSigProc:apa6sigproc6> The 'wiener_threshold_tag' is obsolete, thresholds in summary on 'wiener' tagged traces 
[20:55:35.532] W [sigproc ] <OmnibusSigProc:apa7sigproc7> The 'wiener_threshold_tag' is obsolete, thresholds in summary on 'wiener' tagged traces 
[20:55:35.540] W [sigproc ] <OmnibusSigProc:apa8sigproc8> The 'wiener_threshold_tag' is obsolete, thresholds in summary on 'wiener' tagged traces 
[20:55:35.548] W [sigproc ] <OmnibusSigProc:apa9sigproc9> The 'wiener_threshold_tag' is obsolete, thresholds in summary on 'wiener' tagged traces 
[20:55:35.556] W [sigproc ] <OmnibusSigProc:apa10sigproc10> The 'wiener_threshold_tag' is obsolete, thresholds in summary on 'wiener' tagged traces 
[20:55:35.564] W [sigproc ] <OmnibusSigProc:apa11sigproc11> The 'wiener_threshold_tag' is obsolete, thresholds in summary on 'wiener' tagged traces 
Inputer: "wclsSimDepoSource:"
Outputer: "wclsSimChannelSink:postdrift"
Outputer: "wclsFrameSaver:spsignals"
wclsFrameSaver: promising to produce recob::Wires named "gauss"
wclsFrameSaver: promising to produce recob::Wires named "wiener"
wclsFrameSaver: promising to produce recob::Wires named "dnnsp"
03-Dec-2025 20:55:35 CET  Initiating request to open input file "root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/usertests/a4/f1/000644_reco_data_2025-12-01T_121059Z_reco_data_2025-12-03T_160221Z_reco_data_2025-12-03T_174006Z.root"
03-Dec-2025 20:55:36 CET  Opened input file "root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/usertests/a4/f1/000644_reco_data_2025-12-01T_121059Z_reco_data_2025-12-03T_160221Z_reco_data_2025-12-03T_174006Z.root"
Begin processing the 1st record. run: 20000031 subRun: 0 event: 29 at 03-Dec-2025 20:55:36 CET
SimDepoSource got 5377768 depos from art tag "InputTag: label = 'IonAndScint', instance = ''" returns: okay
SimDepoSource: ready with 5377768 depos spanning: [0.0432409, 2.02045e+06]us
[21:08:19.525] W [  gen   ] <IncoherentAddNoise:addnoiseapa0> model EmpiricalNoiseModel:empericalnoisemeganodes produced 6174 samples instead of expected 6000, future warnings muted 
Retagger: tagging trace set: wiener with 132308 traces, 132308 summary
Retagger: tagging trace set: gauss with 108674 traces, 0 summary
wclsFrameSaver: saving 108674 traces tagged "gauss"
FrameSaver: q=7.20947e+07 n=2274121 tag=gauss
wclsFrameSaver: saving 132308 traces tagged "wiener"
FrameSaver: q=7.90164e+07 n=2091179 tag=wiener
wclsFrameSaver: no traces tagged "dnnsp"
FrameSaver: q=0 n=0 tag=dnnsp
03-Dec-2025 21:49:15 CET  Opened output file with pattern "000644_reco_data_2025-12-01T_121059Z_reco_data_2025-12-03T_160221Z_reco_data_2025-12-03T_174006Z_reco_data_2025-12-03T_195453Z.root"
03-Dec-2025 21:49:27 CET  Closed output file "000644_reco_data_2025-12-01T_121059Z_reco_data_2025-12-03T_160221Z_reco_data_2025-12-03T_174006Z_reco_data_2025-12-03T_195453Z.root"
03-Dec-2025 21:49:27 CET  Closed input file "root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/usertests/a4/f1/000644_reco_data_2025-12-01T_121059Z_reco_data_2025-12-03T_160221Z_reco_data_2025-12-03T_174006Z.root"

================================================================================================================================
TimeTracker printout (sec)                        Min           Avg           Max         Median          RMS         nEvts   
================================================================================================================================
Full event                                      3225.25       3225.25       3225.25       3225.25          0            1     
--------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                        0.00197301    0.00197301    0.00197301    0.00197301         0            1     
simulate:tpcrawdecoder:WireCellToolkit          3218.15       3218.15       3218.15       3218.15          0            1     
simulate:rns:RandomNumberSaver                0.000303717   0.000303717   0.000303717   0.000303717        0            1     
[art]:TriggerResults:TriggerResultInserter    5.2389e-05    5.2389e-05    5.2389e-05    5.2389e-05         0            1     
end_path:out1:RootOutput                      4.5858e-05    4.5858e-05    4.5858e-05    4.5858e-05         0            1     
end_path:out1:RootOutput(write)                 7.10373       7.10373       7.10373       7.10373          0            1     
================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 9016.08 MB
  Peak resident set size usage (VmHWM): 7079.74 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 50 lines of lar log file ===
lar exit code 0
outfile is 000644_reco_data_2025-12-01T_121059Z_reco_data_2025-12-03T_160221Z_reco_data_2025-12-03T_174006Z_reco_data_2025-12-03T_195453Z.root
extractor_prod.py exit code 0
pdjson2metadata exit code 0
Metadata extraction succeeds
===Metadata JSON===
{
    "metadata": {
        "DUNE.campaign": "justIN.w10977s1",
        "art.file_format_era": "ART_2011a",
        "art.file_format_version": 15,
        "art.first_event": 29,
        "art.last_event": 29,
        "art.process_name": "detsim",
        "art.run_type": "physics",
        "core.application": "art.reco",
        "core.application.family": "art",
        "core.application.name": "reco",
        "core.application.version": "v10_11_00d01",
        "core.data_stream": "out1",
        "core.data_tier": "detector-simulated",
        "core.end_time": 1764794967.0,
        "core.event_count": 1,
        "core.file_format": "artroot",
        "core.file_type": "mc",
        "core.first_event_number": 29,
        "core.last_event_number": 29,
        "core.run_type": "physics",
        "core.runs": [
            20000031
        ],
        "core.runs_subruns": [
            2000003100000
        ],
        "core.start_time": 1764794955.0
    },
    "parents": [
        {
            "did": "usertests:000644_reco_data_2025-12-01T_121059Z_reco_data_2025-12-03T_160221Z_reco_data_2025-12-03T_174006Z.root"
        }
    ]
}
===================
.:
total 793932
-rw-r--r--. 1 dune009 dune 812833436 Dec  3 21:49 000644_reco_data_2025-12-01T_121059Z_reco_data_2025-12-03T_160221Z_reco_data_2025-12-03T_174006Z_reco_data_2025-12-03T_195453Z.root
-rw-r--r--. 1 dune009 dune    116698 Dec  3 21:49 wcls-sim-drift-simchannel-nf-sp.log
-rw-r--r--. 1 dune009 dune      8169 Dec  3 21:49 000644_reco_data_2025-12-01T_121059Z_reco_data_2025-12-03T_160221Z_reco_data_2025-12-03T_174006Z_reco_2025-12-03T_195453Z.log
-rw-r--r--. 1 dune009 dune      7101 Dec  3 21:49 jobscript.log
-rw-r--r--. 1 dune009 dune      1141 Dec  3 21:49 000644_reco_data_2025-12-01T_121059Z_reco_data_2025-12-03T_160221Z_reco_data_2025-12-03T_174006Z_reco_data_2025-12-03T_195453Z.root.json
-rw-r--r--. 1 dune009 dune       985 Dec  3 21:49 000644_reco_data_2025-12-01T_121059Z_reco_data_2025-12-03T_160221Z_reco_data_2025-12-03T_174006Z_reco_data_2025-12-03T_195453Z.root.ext.json
-rw-r--r--. 1 dune009 dune       224 Dec  3 20:54 all-input-dids.txt
-rw-r--r--. 1 dune009 dune       193 Dec  3 21:49 justin-processed-pfns.txt
-rw-r--r--. 1 dune009 dune         0 Dec  3 20:55 debugprod.log
justIN time: 2025-12-19 15:44:00 UTC       justIN version: 01.05.03