CL_TEMPLATE
===========

Table of Content
----------------

- `Overview <#overview>`__
- `Features <#features>`__

  - `Design <#design>`__
  - `Verification <#verification>`__
  - `Software <#software>`__
  - `Build <#Build>`__

- `CL Template Quick Start Guide <#cl_template-quick-start-guide>`__

Overview
--------

CL_TEMPLATE is targeted to help customers create a new CustomLogic
example. Developers can update the design, verification, and build flow
to meet their needs without having to tear down a separate example. We
recommend going through other CL examples before creating a new CL.

All of the design files and tests can be compiled, simulated, built, and
deployed on hardware (without any modifications). Developers can
add/update design files, add new verification tests, and create new
build directives to meet their needs.

**To quickly add in design files and test:**

1. Run ``source hdk_setup.sh`` from the top of ``$AWS_FPGA_REPO_DIR/``
2. Run ``cd hdk/cl/examples`` and
   ``./create_new_cl.py --new_cl_name <NAME OF NEW CL>``
3. Run ``cd <NAME OF NEW CL>`` and ``export CL_DIR=$PWD``
4. Place all design files into ``$CL_DIR/design`` (overwriting the
   existing <CL_NAME>.sv file)
5. Simulate with
   ``cd $CL_DIR/verif/scripts and make <CL_NAME>_base_test <SIMULATOR>=1``

Features
--------

Design
~~~~~~

CL_TEMPLATE contains the minimum required set of design files. All
examples require a SystemVerilog file (.sv) named after the CL example
and a Verilog header file (.vh) with required defines.

⚠️ ``cl_id_defines.vh`` is used in the design manifest file generation.
Therefore, this file is required for all CL designs.

`CL_TEMPLATE.sv <https://github.com/aws/aws-fpga/tree/f2/hdk/cl/examples/CL_TEMPLATE/design/CL_TEMPLATE.sv>`__ with the following:

- CL port connections with signal tie downs to avoid simulation errors
- SH_DDR (change ``.EN_DDR(0)`` to ``.EN_DDR(1)`` to elaborate the DDR
  interface)
- ``CL_SH_ID0`` and ``CL_SH_ID1`` connections (see the defines below)

`cl_id_defines.vh <https://github.com/aws/aws-fpga/tree/f2/hdk/cl/examples/cl_sde/design/cl_id_defines.vh>`__ with the following
defines:

- ``CL_SH_ID0``
- ``CL_SH_ID1``

`CL_TEMPLATE_defines.vh <https://github.com/aws/aws-fpga/tree/f2/hdk/cl/examples/CL_TEMPLATE/design/CL_TEMPLATE_defines.vh>`__ with the
following defines:

- ``CL_NAME``

  - See `fpga.sv <https://github.com/aws/aws-fpga/tree/f2/hdk/common/verif/models/fpga/fpga.sv>`__ for
    details

Customers can add all other design files and replace these default files
with their own.

Verification
~~~~~~~~~~~~

All examples use the same SystemVerilog `test
bench <https://github.com/aws/aws-fpga/tree/f2/hdk/common/verif/tb/sv/tb.sv>`__. The common verification
files can be found under
`AWS_FPGA_REPO_DIR/hdk/common/verif <https://github.com/aws/aws-fpga/tree/f2/hdk/common/verif>`__. 
This includes a `common Makefile flow <https://github.com/aws/aws-fpga/tree/f2/hdk/common/verif/tb/scripts>`__ that expects a
`Makefile.tests <https://github.com/aws/aws-fpga/tree/f2/hdk/cl/examples/cl_sde/verif/scripts/Makefile.tests>`__ file that exposes Makefile targets for
each test located under `$CL_DIR/verif/tests <https://github.com/aws/aws-fpga/tree/f2/hdk/cl/examples/cl_sde/verif/tests>`__.

This CL_TEMPLATE example includes a single `base
test <https://github.com/aws/aws-fpga/tree/f2/hdk/cl/examples/CL_TEMPLATE/verif/tests/CL_TEMPLATE_base_test.sv>`__ that will allow all
files to be compiled, elaborated, and quickly simulated to expose any
protocol violations. A `top.SIMULATOR.f <https://github.com/aws/aws-fpga/tree/f2/hdk/cl/examples/cl_sde/verif/scripts/top.xsim.f>`__
file is required for each simulator in use. These file lists are auto
generated by default (including all SystemVerilog source files found
under the `./design <https://github.com/aws/aws-fpga/tree/f2/hdk/common/shell_stable/design>`__ directory). Manual updates are
required for design files or verification infrastructure developed
outside of the `design <https://github.com/aws/aws-fpga/tree/f2/hdk/common/shell_stable/design>`__ or `verif <https://github.com/aws/aws-fpga/tree/f2/hdk/common/verif>`__ directories.
Each example builds on top of the `common file
lists <https://github.com/aws/aws-fpga/tree/f2/hdk/common/verif/tb/filelists>`__.

Required script files:

- `Makefile <https://github.com/aws/aws-fpga/tree/f2/hdk/cl/examples/cl_sde/verif/scripts/Makefile>`__
- `Makefile.tests <https://github.com/aws/aws-fpga/tree/f2/hdk/cl/examples/cl_sde/verif/scripts/Makefile.tests>`__
- `top.xsim.f <https://github.com/aws/aws-fpga/tree/f2/hdk/cl/examples/cl_sde/verif/scripts/top.xsim.f>`__
- `waves.tcl <https://github.com/aws/aws-fpga/tree/f2/hdk/cl/examples/cl_sde/verif/scripts/waves.tcl>`__ (for XSIM only)
- `top.vcs.f <https://github.com/aws/aws-fpga/tree/f2/hdk/cl/examples/cl_sde/verif/scripts/top.vcs.f>`__
- `top.questa.f <https://github.com/aws/aws-fpga/tree/f2/hdk/cl/examples/cl_sde/verif/scripts/top.questa.f>`__

For more information on test bench architecture and functionality:

- Base testbench:
  `$AWS_FPGA_REPO_DIR/hdk/common/verif/tb/sv/tb.sv <https://github.com/aws/aws-fpga/tree/f2/hdk/common/verif/tb/sv/tb.sv>`__
- Basic shell DPI tasks:
  `$AWS_FPGA_REPO_DIR/hdk/common/verif/include/sh_dpi_tasks.svh <https://github.com/aws/aws-fpga/tree/f2/hdk/common/verif/include/sh_dpi_tasks.svh>`__

  - Accessible from the ``tb`` hierarchy
    (``tb.poke_ocl(.addr(64'h100), .data(32'h0)``)

- Basic test tasks:
  `$AWS_FPGA_REPO_DIR/hdk/common/verif/include/common_base_test.svh <https://github.com/aws/aws-fpga/tree/f2/hdk/common/verif/include/common_base_test.svh>`__

  - The `base test <https://github.com/aws/aws-fpga/tree/f2/hdk/cl/examples/CL_TEMPLATE/verif/tests/CL_TEMPLATE_base_test.sv>`__ uses
    ``report_pass_fail_status()``

- Required config files for HBM simulation (not supported in XSIM):
  `$AWS_FPGA_REPO_DIR/hdk/common/verif/include/xpm\_\* <https://github.com/aws/aws-fpga/tree/f2/hdk/common/verif/include/xpm_internal_config_file_sim_1.mem>`__

Co-Simulation with C tests is not yet supported.

Software
~~~~~~~~

CL_TEMPLATE does not yet support stand alone software tests. Please
refer to other examples such as
`CL_DRAM_HBM_DMA <https://github.com/aws/aws-fpga/tree/f2/hdk/cl/examples/cl_dram_hbm_dma/software>`__

Build
~~~~~

The build flow is identical to all other examples:

1. Populate the `constraint files <https://github.com/aws/aws-fpga/tree/f2/hdk/common/shell_stable/build/constraints>`__ as needed
2. Update
   `synth_CL_TEMPLATE.tcl <https://github.com/aws/aws-fpga/tree/f2/hdk/cl/examples/CL_TEMPLATE/build/scripts/synth_CL_TEMPLATE.tcl>`__
   with relevant IP's, constraints, etc.
3. Run `aws_build_dcp_from_cl.py
   <https://github.com/aws/aws-fpga/tree/f2/hdk/common/shell_stable/build/scripts/aws_build_dcp_from_cl.py>`__.

   - Run ``aws_build_dcp_from_cl.py --help`` for more details.

See the following ``tcl`` scripts for more details on the build flow
(they are sourced in
`synth_CL_TEMPLATE.tcl <https://github.com/aws/aws-fpga/tree/f2/hdk/cl/examples/CL_TEMPLATE/build/scripts/synth_CL_TEMPLATE.tcl>`__):

- `$AWS_FPGA_REPO_DIR/hdk/common/shell_stable/build/scripts/synth_cl_header.tcl <https://github.com/aws/aws-fpga/tree/f2/hdk/common/shell_stable/build/scripts/synth_cl_header.tcl>`__
- `$AWS_FPGA_REPO_DIR/hdk/common/shell_stable/build/scripts/synth_cl_footer.tcl <https://github.com/aws/aws-fpga/tree/f2/hdk/common/shell_stable/build/scripts/synth_cl_footer.tcl>`__

Required build scripts:

- `aws_build_dcp_from_cl.py <https://github.com/aws/aws-fpga/tree/f2/hdk/common/shell_stable/build/scripts/aws_build_dcp_from_cl.py>`__
  (symlink) - Python script to run builds
- `build_all.tcl <https://github.com/aws/aws-fpga/tree/f2/hdk/common/shell_stable/build/scripts/build_all.tcl>`__
  (symlink) - Common build script used across examples
- `build_level_1_cl.tcl <https://github.com/aws/aws-fpga/tree/f2/hdk/common/shell_stable/build/scripts/build_level_1_cl.tcl>`__
  (symlink) - Common build script used across examples
- `encrypt.tcl <https://github.com/aws/aws-fpga/tree/f2/hdk/common/shell_stable/build/scripts/encrypt.tcl>`__
  (symlink) - Common build script used across examples to encrypt design
  files
- `synth_CL_TEMPLATE.tcl <https://github.com/aws/aws-fpga/tree/f2/hdk/cl/examples/CL_TEMPLATE/build/scripts/synth_CL_TEMPLATE.tcl>`__ -
  Refer to `STEP 5: Build and Deploy the
  Design <#step-5-build-and-deploy-the-design>`__ below for more
  information.

AWS has integrated basic constraints required across example designs.
These constraint files can be found in
`$AWS_FPGA_REPO_DIR/hdk/common/shell_stable/build/constraints <https://github.com/aws/aws-fpga/tree/f2/hdk/common/shell_stable/build/constraints>`__

Required constraint files:

- `cl_synth_user.xdc <https://github.com/aws/aws-fpga/tree/f2/hdk/cl/examples/cl_sde/build/constraints/cl_synth_user.xdc>`__ - Custom
  constraints applied to the synthesizer (called in
  `synth_CL_TEMPLATE.tcl <https://github.com/aws/aws-fpga/tree/f2/hdk/cl/examples/CL_TEMPLATE/build/scripts/synth_CL_TEMPLATE.tcl>`__)
- `cl_timing_user.xdc <https://github.com/aws/aws-fpga/tree/f2/hdk/cl/examples/cl_sde/build/constraints/cl_timing_user.xdc>`__ -
  Custom timing constraints applied to the synthesizer (called in
  `synth_CL_TEMPLATE.tcl <https://github.com/aws/aws-fpga/tree/f2/hdk/cl/examples/CL_TEMPLATE/build/scripts/synth_CL_TEMPLATE.tcl>`__)
- `small_shell_cl_pnr_user.xdc <https://github.com/aws/aws-fpga/tree/f2/hdk/cl/examples/cl_sde/build/constraints/small_shell_cl_pnr_user.xdc>`__
  - Floorplan constraints for the CL design when building with
  SMALL_SHELL. See `CL_DRAM_HBM_DMA's
  constraints <https://github.com/aws/aws-fpga/tree/f2/hdk/cl/examples/cl_dram_hbm_dma/build/constraints/small_shell_cl_pnr_user.xdc>`__
  for reference.
- `xdma_shell_cl_pnr_user.xdc <https://github.com/aws/aws-fpga/tree/f2/hdk/cl/examples/cl_sde/build/constraints/xdma_shell_cl_pnr_user.xdc>`__
  - Floorplan constraints for the CL design when building with
  XDMA_SHELL. See `CL_DRAM_HBM_DMA's
  constraints <https://github.com/aws/aws-fpga/tree/f2/hdk/cl/examples/cl_dram_hbm_dma/build/constraints/xdma_shell_cl_pnr_user.xdc>`__
  for reference.

For more information on how to populate the constraint files and build
scripts, please refer to other examples such as
`CL_DRAM_HBM_DMA <https://github.com/aws/aws-fpga/tree/f2/hdk/cl/examples/cl_dram_hbm_dma/build>`__.

CL_TEMPLATE Quick Start Guide
-----------------------------

The HDK top level `Quick Start Guide <./../../../README.html#getting-started>`__
will provide an introduction to this guide.

STEP 1: Create a New CL
~~~~~~~~~~~~~~~~~~~~~~~

The first step is to copy this example into a new directory and replace all
references to CL_TEMPLATE with the new example name. To automatically create
your own example, run `create_new_cl.py <https://github.com/aws/aws-fpga/tree/f2/hdk/cl/examples/create_new_cl.py>`__ from the
``$AWS_FPGA_REPO_DIR/hdk/cl/examples`` directory

.. code:: bash

  # CL names are typically all lowercase
  export NEW_CL_NAME='<new cl name>'
  cd $AWS_FPGA_REPO_DIR/hdk/cl/examples
  ./create_new_cl.py --new_cl_name ${NEW_CL_NAME}

Or run these shell commands

.. code:: bash

  export NEW_CL_NAME='<new cl name>'

  # Everything below can be copied and pasted into a bash terminal
  CL_TEMPLATE=CL_TEMPLATE
  echo "Creating $CL_TEMPLATE directory"
  cp -r $CL_TEMPLATE $NEW_CL_NAME

  echo "Replacing CL_TEMPLATE with $CL_TEMPLATE in all files"
  grep -rl $CL_TEMPLATE $NEW_CL_NAME | xargs sed -i "s/$CL_TEMPLATE/$NEW_CL_NAME/g"

  echo "Updating CL_TEMPLATE with $CL_TEMPLATE in all file names"
  CL_TEMPLATE=$CL_TEMPLATE NEW_CL_NAME=$NEW_CL_NAME find $NEW_CL_NAME -name "*$CL_TEMPLATE*" -exec sh -c 'mv "$0" "${$0/$CL_TEMPLATE/$NEW_CL_NAME}"' {} \;

The result will be a directory containing everything in CL_TEMPLATE
(renamed to the new CL name) with functional tools and scripts. Please
note that these files need to be manually updated to include any design
specific changes.

STEP 2: Add Design Files
~~~~~~~~~~~~~~~~~~~~~~~~

All CL design files should be placed under the `design <./design>`_ directory.
Please see the `Design <#design>`__ section for information on the CL_TEMPLATE
design files.

Users may modify these files and add new ones as their designs grow.
Xilinx IP's are available to all CL examples (found under
`$AWS_FPGA_REPO_DIR/hdk/common/ip <https://github.com/aws/aws-fpga/tree/f2/hdk/common/ip>`__).

STEP 3: Develop Design Verification Tests (OPTIONAL)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Once design files have been added, run the
`CL_TEMPLATE_base_test <https://github.com/aws/aws-fpga/tree/f2/hdk/cl/examples/CL_TEMPLATE/verif/tests/CL_TEMPLATE_base_test.sv>`__. This
test only powers up the test bench to make sure the design can be
compiled and simulated. All tests should be located under
`$CL_DIR/verif/tests <https://github.com/aws/aws-fpga/tree/f2/hdk/cl/examples/cl_sde/verif/tests>`__ and test targets added to
`Makefile.tests <https://github.com/aws/aws-fpga/tree/f2/hdk/cl/examples/cl_sde/verif/scripts/Makefile.tests>`__:

.. code:: bash

  cd $AWS_FPGA_REPO_DIR/hdk/cl/examples/CL_TEMPLATE
  export CL_DIR=$(pwd)
  cd ${CL_DIR}/verif/scripts
  make CL_TEMPLATE_base_test

Or specify an `available simulator
<./../../../../User_Guide_AWS_EC2_FPGA_Development_Kit.html#hardware-development-kit-hdk>`__:

.. code:: bash

  make CL_TEMPLATE_base_test <SIMULATOR>=1

This will first generate and compile the simulation libraries required for the
requested SIMULATOR. Test results will be stored in the
``$CL_DIR/verif/sim/<SIMULATOR>`` directory (created upon first simulation
run). After adding new IP's to
`$AWS_FPGA_REPO_DIR/hdk/common/ip <https://github.com/aws/aws-fpga/tree/f2/hdk/common/ip>`__, the simulation
libraries need to be recompiled: ``make regenerate_sim_libs <SIMULATOR>=1``.

File List Generation
^^^^^^^^^^^^^^^^^^^^

The Makefile includes ``$AWS_FPGA_REPO_DIR/hdk/common/verif/tb/scripts/Makefile.common.inc``
which runs a Python script to automatically update each ``top.<SIMULATOR>.f`` file list with
all SystemVerilog files found under the ``$CL_DIR/design/`` directory.

- To disable the generation, run ``export DONT_GENERATE_FILE_LIST=1``.
- To re-enable the generation, run ``unset DONT_GENERATE_FILE_LIST``.
- To generate the file list by itself, run ``make generate_sim_file_list <SIMULATOR>=1``.
- To add additional files, add them outside of the auto generation section:

.. code:: text

  # Add code up here or below the comment block to persist between simulations

  ##############################
  #### BEGIN AUTO-GENERATE #####

  +incdir+$CL_DIR/design/

  $CL_DIR/design/CL_TEMPLATE.sv

  ##### END AUTO-GENERATE ######
  ##############################

Xilinx/AMD IP Discovery and Compilation
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

When running your first test, all Xilinx IP's under
`$AWS_FPGA_REPO_DIR/hdk/common/ip/cl_ip <https://github.com/aws/aws-fpga/tree/f2/hdk/common/ip/cl_ip>`__
are automatically compiled

- You can find the ``xil_defaultlib`` library for each simulator in
  `$AWS_FPGA_REPO_DIR/hdk/common/verif/ip_simulation_libraries/
  <https://github.com/aws/aws-fpga/tree/f2/hdk/common/verif>`__ (created after first simulation run)

If a design adds new IP's, make sure to add the new simulation libraries
to ``COMMON_LIBLISTS`` in:

- `$AWS_FPGA_REPO_DIR/hdk/common/verif/tb/scripts/Makefile.common.inc
  <https://github.com/aws/aws-fpga/tree/f2/hdk/common/verif/tb/scripts/Makefile.common.inc>`__ (this is
  required for XSIM and Questa simulations).

Simulation library names can be found under:

- `$AWS_FPGA_REPO_DIR/hdk/common/ip/cl_ip/cl_ip.ip_user_files/sim_scripts
  <https://github.com/aws/aws-fpga/tree/f2/hdk/common/ip/cl_ip/cl_ip.ip_user_files/sim_scripts>`__ followed
  by ``<IP_NAME>/<SIMULATOR>/<IP_NAME>.sh``

All verification work is located under the `verif <https://github.com/aws/aws-fpga/tree/f2/hdk/common/verif>`__
directory. Please see the `Verification <#verification>`__ section for
the CL_TEMPLATE verification details.

STEP 4: Develop Software Tests (OPTIONAL)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

CL_TEMPLATE does not yet support standalone software tests. Please
refer to other examples such as `CL_DRAM_HBM_DMA <https://github.com/aws/aws-fpga/tree/f2/hdk/cl/examples/cl_dram_hbm_dma/software>`__

STEP 5: Build and Deploy the Design
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Once design files have been added and tested, constraint and build
script updates need to be made. The build flow creates a bitstream that
is used to create an AFI to deploy on hardware. The build and deployment
flows can be found in the `Quick Start Guide
<./../../../README.html#getting-started>`__. Examples for all constraint files
and build scripts can be found in other examples such as
`CL_DRAM_HBM_DMA <https://github.com/aws/aws-fpga/tree/f2/hdk/cl/examples/cl_dram_hbm_dma/build>`__.

1. Populate existing and/or add new constraint files to
   `./build/constraints <https://github.com/aws/aws-fpga/tree/f2/hdk/common/shell_stable/build/constraints>`__

   - AWS has provided basic constraints required in example designs in
     `$AWS_FPGA_REPO_DIR/hdk/common/shell_stable/build/constraints <https://github.com/aws/aws-fpga/tree/f2/hdk/common/shell_stable/build/constraints>`__

2. Update
   `synth_CL_TEMPLATE.tcl <https://github.com/aws/aws-fpga/tree/f2/hdk/cl/examples/CL_TEMPLATE/build/scripts/synth_CL_TEMPLATE.tcl>`__
   with:

   - Additional constraint files to be applied during a build
   - Additional tcl scripts to synthesize the design
   - Xilinx IP's utilized in the design

     - IP's used in AWS example designs can uncommented be in
       `synth_CL_TEMPLATE.tcl <https://github.com/aws/aws-fpga/tree/f2/hdk/cl/examples/CL_TEMPLATE/build/scripts/synth_CL_TEMPLATE.tcl>`__
       and found under
       `$AWS_FPGA_REPO_DIR/hdk/common/ip/cl_ip <https://github.com/aws/aws-fpga/tree/f2/hdk/common/ip/cl_ip>`__
     - If a design adds new IP's, make sure to add the new ``xci`` files
       to the build script
       `$CL_DIR/build/scripts/synth_CL_TEMPLATE.tcl <https://github.com/aws/aws-fpga/tree/f2/hdk/cl/examples/CL_TEMPLATE/build/scripts/synth_CL_TEMPLATE.tcl>`__

3. Run `aws_build_dcp_from_cl.py <https://github.com/aws/aws-fpga/tree/f2/hdk/common/shell_stable/build/scripts/aws_build_dcp_from_cl.py>`__.
   All defaults can be found by running ``aws_build_dcp_from_cl.py --help``

  .. code:: bash

    cd  $AWS_FPGA_REPO_DIR/hdk/cl/examples/CL_TEMPLATE
    export CL_DIR=$(pwd)
    cd ${CL_DIR}/build/scripts
    ./aws_build_dcp_from_cl.py -c CL_TEMPLATE

  The start of the log will provide build details:

  .. code:: bash

    ==================================================
    Running CL builds
    ==================================================
    cl              : cl_mem_perf
    mode            : xdma_shell
    clock_recipe_a  : A1
    clock_recipe_b  : B2
    clock_recipe_c  : C0
    clock_recipe_hbm : H2
    flow            : BuildAll
    place_direct    : SSI_ExtraTimingOpt
    phy_opt_direct  : Explore
    route_direct    : AggressiveExplore
    build_tag       : None
    ==================================================
    vivado -mode batch -source build_all.tcl -log 04_06_2023-181511_build_all.log -tclargs SSI_ExtraTimingOpt Explore AggressiveExplore A1 B2 C0 H2

  The build will create 4 new subdirectories under `./build <https://github.com/aws/aws-fpga/tree/f2/hdk/common/shell_stable/build>`__:
    - ``src_post_encryption`` - encrypted design files
    - ``checkpoints`` - design checkpoints (``.dcp``) for each stage used by
      Vivado
    -  ``reports`` - reports for each build stage
    - ``bitstreams`` - CustomLogic bitstreams used for AFI creation

  Builds with timing violations will have a suffix of ``_VIOLATED`` under the
  ``checkpoints`` directory. Details can be found under the ``reports``
  directory.

After a successful build, you can follow the AFI creation, loading, and
testing instructions in the `Hardware Development Kit (HDK) top level document
<./../../../README.html#build-accelerator-afi-using-hdk-design-flow>`__.