Standalone例程的应用与分析

如果是VCK190 ES单板，需要在Lounge里申请"Versal Tools Early Eacess"; "Versal Tools PDI Early Eacess"的License，并在Vivado里使能ES器件。在Vivado/2020.2/scripts/init.tcl的文件里，添加“enable_beta_device xcvc*”，可以自动使能ES器件。

1.2. Platform
在进行开发之前，需要准备Platform。 VCK190 Production单板和VCK190 ES单板使用的Platform不一样，可以从下面链接下载各自的Platform，再复制到目录“Xilinx/Vitis/2020.2/platforms/”下。

VCK190 Production Platform
VCK190 ES Platform

准备好后，目录结构与下面类似。

1.3. Common Images

Xilinx现在还提供了Common Images，包含对应单板的Linux启动文件，和编译器、sysroots（头文件、应用程序库）等。可以在Xilinx Download下载Versal common image。

1.4. 测试环境
Host OS： Ubuntu 18.04
Vitis 2020.2
PetaLinux 2020.2
VCK190 Production

2. AIE a2z 分析
2.1. 介绍

例程AIE a2z是Standalone （BareMetal）的例程，Versal的A72不运行Linux。 它很全面，包含创建Platform、创建AIE Kernel、创建PL Kernel、创建A72应用程序、调试AIE Kernel。在Xilinx的文档中，AIE的程序，叫Kernel； 在Vitis里使用HLS开发的PL设计，也叫Kernel。

注意，2021年7月份，Vitis Tutorials的"master"分支，才包含例程AIE a2z。

2.2. 文件列表
AIE a2z包含下列文件。

后缀是md的文件，详细说了完成AIE a2z例子各个步骤。

AI_Engine_Development/Feature_Tutorials/01-aie_a_to_z$ ls -l
01-custom_base_platform_creation.md
02-aie_application_creation.md
03-pl_application_creation.md
04-ps_application_creation_run_all.md
images
LICENSE.txt
README.md
script
src

src目录包含源代码文件，分别对应A72、PL、AIE设计。
AI_Engine_Development/Feature_Tutorials/01-aie_a_to_z/src$ tree
main.cpp
mm2s.cpp
platform_config.h
platform.cpp
platform.h
s2mm.cpp

2.3. main.cpp
main.cpp是A72的主要代码。除去Standalone软件的系统初始化，它还完成以下工作。

1. 分别申请内存，初始化输入输出数据
2. 通过AIE_systemConfig启动MM2S/S2MM的数据搬移
3. 初始化和运行Graph
4. 检查输出数据是否正确

int main()
{

printf("Initializing input data in memory\n");
status = InitInputData(&in, INPUT_SIZE);

printf("Initializing output memory space\n");
status = InitOutputData(&out, OUTPUT_SIZE);

printf("System Configuration\n");
AIE_systemConfig(in, out, INPUT_SIZE, OUTPUT_SIZE);

printf("Graph Initialization\n");
mygraph.init();

printf("Running Graph for 4 iterations\n");
mygraph.run(4);

int checks = 1;

while(1) {
printf("Checking output, check #%d\n",checks);
uint32_t v = Xil_In32(S2MM_BASE + CTRL_OFFSET);
checks++;
if(v & 6) {
break;
}
sleep(2);
}

printf("Checking Output Data: \n");
int err = 0;
int32_t golden, dataIn, realTemp, imTemp;
for(int i = 0; i dataIn = ((int32_t*)in)[i];
realTemp = (dataIn & 0xFFFF) + ((dataIn & 0xFFFF0000)>>16);
imTemp = (dataIn & 0xFFFF) - ((dataIn & 0xFFFF0000)>>16);
golden = ((realTemp - imTemp)

if(golden != ((int32_t*)out)[i])
{
err++;
printf("Output Error: Golden = %x Output = %x\n ",golden,((int32_t*)out)[i]);
}
}
}

2.4. mm2s.cpp
mm2s.cpp是利用HLS做的PL设计，用于从内存搬移数据到AIE Kernel。
extern "C" {
void mm2s(ap_int* mem, hls::stream >& s, int size) {
#pragma HLS INTERFACE m_axi port=mem offset=slave bundle=gmem

#pragma HLS interface axis port=s

#pragma HLS INTERFACE s_axilite port=mem bundle=control
#pragma HLS INTERFACE s_axilite port=size bundle=control
#pragma HLS interface s_axilite port=return bundle=control

for(int i = 0; i #pragma HLS PIPELINE II=1
qdma_axis x;
x.data = mem[i];
x.keep_all();
s.write(x);
}
}
}

2.5. s2mm.cpp
mm2s.cpp也是利用HLS做的PL设计，用于从AIE Kernel搬移数据到内存。
extern "C" {

void s2mm(ap_int* mem, hls::stream >& s, int size) {
#pragma HLS INTERFACE m_axi port=mem offset=slave bundle=gmem

#pragma HLS interface axis port=s

#pragma HLS INTERFACE s_axilite port=mem bundle=control
#pragma HLS INTERFACE s_axilite port=size bundle=control
#pragma HLS interface s_axilite port=return bundle=control

for(int i = 0; i #pragma HLS PIPELINE II=1
qdma_axis x = s.read();
mem[i] = x.data;
}
}
}

2.6. AIE工程文件
AIE a2z例子中，没有提供AIE的代码。AIE代码，来自于Vitis中AIE的模板工程“Simple”。AIE的模板工程“Simple”包含文件kernels.h、kernels.cc、project.h、project.cpp。

kernels.cc是定义AIE Kernel的文件，也是最重要的文件，它实际运行在AIE上。

void simple(input_window_cint16 * in, output_window_cint16 * out) {
cint16 c1, c2;
for (unsigned i=0; i window_readincr(in, c1);
c2.real = c1.real+c1.imag;
c2.imag = c1.real-c1.imag;
window_writeincr(out, c2);
}
}

kernels.h声明了AIE Kernel函数的的原型。

void simple(input_window_cint16 * in, output_window_cint16 * out);

project.h定义了运算的graph，连接了stream数据流和AIE kernel。

class simpleGraph : public adf::graph {
private:
kernel first;
kernel second;
public:
input_port in;
output_port out;
simpleGraph(){

first = kernel::create(simple);
second = kernel::create(simple);
connect > net0 (in, first.in[0]);
connect > net1 (first.out[0], second.in[0]);
connect > net2 (second.out[0], out);

source(first) = "kernels/kernels.cc";
source(second) = "kernels/kernels.cc";

runtime(first) = 0.1;
runtime(second) = 0.1;

}
};

project.cpp定义和控制运算的graph。A72的应用程序，包含了project.cpp，并且执行了mygraph.init()和mygraph.run( )。

simpleGraph mygraph;

# if 0
simulation::platform platform("data/input.txt", "data/output.txt");
# else
PLIO *in0 = new PLIO("DataIn1", adf::plio_32_bits,"data/input.txt");
PLIO *out0 = new PLIO("DataOut1",adf::plio_32_bits, "data/output.txt");
simulation::platform platform(in0, out0);
# endif

connect net0(platform.src[0], mygraph.in);
connect net1(mygraph.out, platform.sink[0]);

# ifdef __AIESIM__

int main(void) {
mygraph.init();
mygraph.run(4);
mygraph.end();
return 0;
}
# endif // __AIESIM__

3. 经验

AIE a2z做得相当完善，基本可以顺利完成。 在实验过程中，可能遇到下列问题。

3.1. AXI Interrupt
创建平台（Platform）时，AXI中断控制器（axi_intc）没有连接中断源。Vitis编译工程时，会连接HLS设计的IP模块的中断输出到AXI中断控制器（axi_intc）。 如果验证平台（Platform）的Block Design时，Vivado会报告下列关于中断控制器消息，提示没有中断源，可以忽略。

[BD 41-759] The input pins (listed below) are either not connected or do not have a source port, and they don't have a tie-off specified. These pins are tied-off to all 0's to avoid error in Implementation flow.
Please check your design and connect them as needed:
/axi_intc_0/intr

3.2. sys_clk0
Vivado也会对输入时钟报告下列时钟不匹配的消息。Vivado创建Block Design时，默认的时钟是100MHz。单板上的实际时钟是200MHz。选中sys_clk0_0，在属性中，把它更改为200MHz。

[xilinx.com:ip:axi_noc:1.0-1] /ps_nocClock frequency of the connected clock (/ps_noc/sys_clk0) is 100.000 MHz while "Input System Clock Frequency" is 200.000 MHz. Please either reconfigure the parameter "Input System Clock Period" of the axi_noc (in DDR Basic tab) or change frequency of the connected clock (CONFIG.FREQ_HZ) within the range of 199920031.987 to 200080032.013 Hz.

3.3. AIE license
如果Vitis编译工程时，报告“AIE license not found”，请申请license。

AIE license not found !
/opt/Xilinx/Vitis/2020.2/aietools/bin/aieir_be: line 96: kill: (-28000) - No such process
ERROR: [aiecompiler 77-753] This application has discovered an exceptional condition from which it cannot recover while executing the following command
>> aieir_be --time-passes=0 --trace-plio-width=64 --pl-freq=0 --use-real-noc=true --show-loggers=false --high-performance=false --kernel-address-location=false --target=x86sim --swfifo-threshold=40 --single-mm2s-channel=false --workdir=./Work --exit-after=complete --event-trace-config= --test-iterations=-1 --stacksize=1024 --platform=/proj/hankf/vck190/vck190_aie_a2z/vitis/base_pfm_vck190_aie_a2z/export/base_pfm_vck190_aie_a2z/base_pfm_vck190_aie_a2z.xpfm --event-trace-custom-config= --disable-dma-cmd-alignment=false --enable-ecc-scrubbing=false --write-partitioned-file=true --schemafile=AIEGraphSchema.json --includ --includ --includ --includ --includ --includ --device= --write-unified-data=false --fastmath=false --event-trace-advanced-mapping=0 --log-level=1 --enable-reconfig=false --aiesim-xrt-api=false --gen-graph-cleanup=false --use-canonical-net-names=false --event-trace-port=plio --new-placer=true --use-phy-shim=true --xlopt=0 --pre-compile-kernels=false --validate-only=false --trace-aiesim-option=0 --aiearch=aie --mapped-soln-udm= --optimize-pktids=false --no-init=false --num-trace-streams=1 --aie-heat-map=false --phydevice= --exec-timed=0 --pl-auto-restart=false --routed-soln-udm= --enable-profiling=false --disable-transform-merge-broadcast=false --verbose=true --use-async-rtp-locks=true --repo-path= --genArchive=false --pl-axi-lite=false --new-router=true --aie-driver-v1=false --logcfg-file= --event-trace-bounding-box= --enable-reconfig-dma-autostart=false --heapsize=1024 --logical-arch= --nodot-graph=false --shim-constraints= --disable-dma-autostart=false --disable-transform-broadcast-split=true -json ./Work/temp/project.json -sdf-graph /proj/hankf/vck190/vck190_aie_a2z/vitis/simple_application_vck190_aie_a2z/src/project.cpp.

3.4. 安装dot
Vitis在编译过程中，会用到工具dot。如果没有安装sudo apt install graphviz，会得到错误"sh: 1: dot: not foun"。 在Ubuntu 18.04下，如果有管理员权限，使用命令“sudo apt install graphviz”能安装dot。

DEBUG:MapperPartitioner: Adding Edge : Name=D_net2 SrcPort=i1_po0 DstPort=i3_pi0 EdgeType=mem
DEBUG:MapperPartitioner:Done--Add Double Buffer Edge SrcPort=i1_po0 DstPort=i3_pi0 type=mem Edge=net2:i1-(buf2)->i3
DEBUG:MapperPartitioner:Graph After Adding Double Edges
sh: 1: dot: not found

ERROR: [aiecompiler 77-753] This application has discovered an exceptional condition from which it cannot recover while executing the following command
>> dot ./Work/reports/project.dot -Tpng -o ./Work/reports/project.png
.
Please check the output log for errors and fix those before you run the application.
/opt/Xilinx/Vitis/2020.2/aietools/bin/aieir_be: line 96: kill: (-44668) - No such process
ERROR: [aiecompiler 77-753] This application has discovered an exceptional condition from which it cannot recover while executing the following command
>> aieir_be --time-passes=0 --trace-plio-width=64 --pl-freq=0 --use-real-noc=true --show-loggers=false --high-performance=false --kernel-address-location=false --target=hw --swfifo-threshold=40 --single-mm2s-channel=false --workdir=./Work --exit-after=complete --event-trace-config= --test-iterations=-1 --stacksize=1024 --platform=/proj/hankf/vck190/vck190_aie_a2z/vitis/base_pfm_vck190_aie_a2z/export/base_pfm_vck190_aie_a2z/base_pfm_vck190_aie_a2z.xpfm --event-trace-custom-config= --disable-dma-cmd-alignment=false --enable-ecc-scrubbing=false --write-partitioned-file=true --schemafile=AIEGraphSchema.json --includ --includ --includ --includ --includ --includ --device= --write-unified-data=false --fastmath=false --event-trace-advanced-mapping=0 --log-level=1 --enable-reconfig=false --aiesim-xrt-api=false --gen-graph-cleanup=false --use-canonical-net-names=false --event-trace-port=plio --new-placer=true --use-phy-shim=true --xlopt=0 --pre-compile-kernels=false --validate-only=false --trace-aiesim-option=0 --aiearch=aie --mapped-soln-udm= --optimize-pktids=false --no-init=false --num-trace-streams=1 --aie-heat-map=false --phydevice= --exec-timed=0 --pl-auto-restart=false --routed-soln-udm= --enable-profiling=false --disable-transform-merge-broadcast=false --verbose=true --use-async-rtp-locks=true --repo-path= --genArchive=false --pl-axi-lite=false --new-router=true --aie-driver-v1=false --logcfg-file= --event-trace-bounding-box= --enable-reconfig-dma-autostart=false --heapsize=1024 --logical-arch= --nodot-graph=false --shim-constraints= --disable-dma-autostart=false --disable-transform-broadcast-split=true -json ./Work/temp/project.json -sdf-graph /proj/hankf/vck190/vck190_aie_a2z/vitis/simple_application_vck190_aie_a2z/src/project.cpp.

3.5. 软件Emulation
运行软件Emulation的时候，要选择AIE工程，不选择system project。如果选择system project运行软件Emulation，会出现下列错误。

Error while launching program:
The selected system project 'simple_application_system' contains applications (simple_application) that doesn't support launching software emulation.
The selected system project 'simple_application_system' contains applications (simple_application) that doesn't support launching software emulation.

3.6. 硬件Emulation
先运行软件Emulation，再运行硬件Emulation。
如果直接运行硬件Emulation，会出现下列错误。

Failed to start emulator on the project 'simple_application_system' using the build configuration 'Emulation-HW'.
Launch emulator script doesn't exist at location '/proj/hankf/vck190/vck190_aie_a2z_script_hw_prj/custom_pfm_vck190/vitis/simple_application_system/Emulation-HW/package/launch_hw_emu.sh'.

另外Vitis里，先选择AIE工程，再编译AIE工程，然后去启动硬件Emulation，菜单里可能没有目标。编译后，要重新选择system project，再选择AIE工程，再去启动硬件Emulation，菜单里就会有目标。

3.7. A72软件没有ap_int.h
文件mm2s.cpp和s2mm.cpp时给HLS设计用的，不能添加到A72的软件工程里。如果把它们加到了A72的软件工程里，会遇到错误“ap_int.h: No such file or directory”。

aarch64-none-elf-g++ -Wall -O0 -g3 -I"/opt/Xilinx/Vitis/2020.2/aietools/include"
-I"/proj/hankf/vck190/vck190_aie_a2z/vitis/simple_application_vck190_aie_a2z/src"
-I"/../include" -c -fmessage-length=0 -MT"src/mm2s.o" -mcpu=cortex-a72 -I/proj/hankf/vck190/vck190_aie_a2z/vitis/vck190_aie_a2z_aie_output_platform/export/vck190_aie_a2z_aie_output_platform/sw/vck190_aie_a2z_aie_output_platform/standalone_domain/bspinclude/include -MMD -MP -MF"src/mm2s.d" -MT"src/mm2s.o" -o "src/mm2s.o" "../src/mm2s.cpp"
../src/mm2s.cpp:33:10: fatal error: ap_int.h: No such file or directory
33 | #include
| ^~~~~~~~~~

3.8. A72软件工程找不到simple(input_window, output_window)
A72软件要控制AIE Kernel，需要相关信息。因此预先把AIE工程编译后产生的文件“Hardware/Work/ps/c_rts/aie_control.cpp“，添加到 A72软件工程。

如果忘记添加，可能会得到错误信息，“undefined reference to `simple(input_window, output_window)'”

aarch64-none-elf-g++ -L/opt/Xilinx/Vitis/2020.2/aietools/lib/aarchnone64.o -mcpu=cortex-a72 -Wl,-T -Wl,../src/lscript.ld -L/proj/hankf/vck190/vck190_aie_a2z/vitis/vck190_aie_a2z_aie_output_platform/export/vck190_aie_a2z_aie_output_platform/sw/vck190_aie_a2z_aie_output_platform/standalone_domain/bsplib/lib -o "aie_a2z_vck190_a72_ctrl_app.elf" ./src/main.o ./src/platform.o -ladf_api -Wl,--start-group,-lxil,-lgcc,-lc,-lstdc++,--end-group
/opt/Xilinx/Vitis/2020.2/gnu/aarch64/lin/aarch64-none/x86_64-oesdk-linux/usr/bin/aarch64-xilinx-elf/aarch64-xilinx-elf-ld.real: ./src/main.o: in function `simpleGraph::simpleGraph()':
/proj/hankf/vck190/vck190_aie_a2z/vitis/simple_application_vck190_aie_a2z/src/project.h:17: undefined reference to `simple(input_window*, output_window*)'
/opt/Xilinx/Vitis/2020.2/gnu/aarch64/lin/aarch64-none/x86_64-oesdk-linux/usr/bin/aarch64-xilinx-elf/aarch64-xilinx-elf-ld.real: /proj/hankf/vck190/vck190_aie_a2z/vitis/simple_application_vck190_aie_a2z/src/project.h:17: undefined reference to `simple(input_window*, output_window*)'
/opt/Xilinx/Vitis/2020.2/gnu/aarch64/lin/aarch64-none/x86_64-oesdk-linux/usr/bin/aarch64-xilinx-elf/aarch64-xilinx-elf-ld.real: /proj/hankf/vck190/vck190_aie_a2z/vitis/simple_application_vck190_aie_a2z/src/project.h:18: undefined reference to `simple(input_window*, output_window*)'
makefile:48: recipe for target 'aie_a2z_vck190_a72_ctrl_app.elf' failed
/opt/Xilinx/Vitis/2020.2/gnu/aarch64/lin/aarch64-none/x86_64-oesdk-linux/usr/bin/aarch64-xilinx-elf/aarch64-xilinx-elf-ld.real: /proj/hankf/vck190/vck190_aie_a2z/vitis/simple_application_vck190_aie_a2z/src/project.h:18: undefined reference to `simple(input_window*, output_window*)'
collect2.real: error: ld returned 1 exit status
make: *** [aie_a2z_vck190_a72_ctrl_app.elf] Error 1

3.9. Package

编译A72程序后，要编译system project，将所有模块打包再一起。这时候，要根据04-ps_application_creation_run_all.md的Step 3. Build the Full System，添加打包选项，“--package.ps_elf ../../A-to-Z_app/Debug/A-to-Z_app.elf,a72-0 --package.defer_aie_run”。

如果没有添加，会报告错误“no xclbin input is found”。

Package step cannot be performed since the platform has a VPP link generated XSA and no xclbin input is found. Please provide a valid xclbin location in system project package options
11:21:21 Build Finished (took 646ms)