開源100 Gbps NIC Corundum環(huán)境的搭建

一、Corundum簡(jiǎn)介

GitHub repository: https://github.com/corundum/corundum

GitHub wiki: https://github.com/corundum/corundum/wiki

Corundum是一個(gè)基于FPGA的開源原型平臺(tái)，用于高達(dá)100Gbps及更高的網(wǎng)絡(luò)接口開發(fā)。Corundum平臺(tái)包括一些用于實(shí)現(xiàn)實(shí)時(shí)，高線速操作的核心功能，包括：高性能數(shù)據(jù)路徑，10G/ 25G / 100G以太網(wǎng)MAC，PCIExpress第3代，自定義PCIeDMA引擎以及本機(jī)高精確的IEEE 1588 PTP時(shí)間戳。一個(gè)關(guān)鍵功能是可擴(kuò)展隊(duì)列管理，它可以支持超過10,000個(gè)隊(duì)列以及可擴(kuò)展的傳輸調(diào)度程序，從而可以對(duì)包傳輸進(jìn)行細(xì)粒度的硬件控制。結(jié)合多個(gè)網(wǎng)絡(luò)接口，每個(gè)接口多個(gè)端口以及每個(gè)端口事件驅(qū)動(dòng)的傳輸調(diào)度，這些功能可實(shí)現(xiàn)高級(jí)網(wǎng)絡(luò)接口，體系結(jié)構(gòu)和協(xié)議的開發(fā)。這些硬件功能的軟件接口是Linux網(wǎng)絡(luò)協(xié)議棧的高性能驅(qū)動(dòng)程序。該平臺(tái)還支持分散/聚集DMA，校驗(yàn)和卸載，接收流散列和接收端縮放。一個(gè)全面的，基于Python的開放源代碼仿真框架促進(jìn)了開發(fā)和調(diào)試，該框架包括整個(gè)系統(tǒng)，從驅(qū)動(dòng)程序和PCIExpress接口的仿真模型到以太網(wǎng)接口。通過實(shí)現(xiàn)微秒級(jí)時(shí)分多址（TDMA）硬件調(diào)度程序，以100Gbps的線速執(zhí)行TDMA調(diào)度，而沒有CPU開銷，證明了Corundum的強(qiáng)大功能和靈活性。

二、環(huán)境準(zhǔn)備

2.1：Linux系統(tǒng)

本機(jī)為win10專業(yè)版，由于環(huán)境需要

在VMware虛擬機(jī)安裝Ubuntu 20.04 LTS系統(tǒng)供日常學(xué)習(xí)測(cè)試使用

安裝python庫(kù)

使用tox嘗試仿真

安裝雙系統(tǒng)Ubuntu 20.04 LTS

在Ubuntu系統(tǒng)上安裝vivado 2019.2

注意vivado 2019.2的license，基本組件的license會(huì)編譯出錯(cuò)，需要添加IP license。

2.2：Obtaining the source code

直接從git地址下載zip壓縮包使用；

$ git clone https://github.com/corundum/corundum.git

2.3：FPGA仿真環(huán)境搭建
2.3.1 system packages
Python3.9；當(dāng)前ubuntu 20.04自帶python3版本為python3.8，需要下載python3.9并配置系統(tǒng)python3為python3.9。其中遇到很多問題需要百度自行解決。。。

$ sudo apt-get install python3.9
$ sudo update-alternatives --install /usr/bin/python3 python3 /usr/bin/python3.8 1

$ sudo update-alternatives --install /usr/bin/python3 python3 /usr/bin/python3.9 2
$ sudo update-alternatives --config python3
There are 2 choices for the alternative python3 (providing /usr/bin/python3).

Selection Path Priority Status
------------------------------------------------------------
* 0 /usr/bin/python3.8 2 auto mode
1 /usr/bin/python3.8 2 manual mode
2 /usr/bin/python3.9 1 manual mode

Press to keep the current choice[*], or type selection number: 2

測(cè)試python版本：

$ python3 -V
Python 3.9

安裝/升級(jí)pip

$ sudo apt-get install python3-pip
$ sudo pip3 install --upgrade pip

安裝python-dev

$ sudo apt install python3-dev

安裝Icarus Verilog

$ sudo apt-get install iverilog

安裝GTKWave

$ sudo apt-get install gtkwave

2.3.2 required python packages

cocotb

cocotb-bus

cocotb-test

cocotbext-axi

cocotbext-eth

cocotbext-pcie

pytest

scapy

Recommended additional python packages:

tox (to run pytest inside a python virtual environment)

pytest-xdist (to run tests in parallel with pytest -n auto)

pytest-sugar (makes pytest output a bit nicer)

$ pip3 install cocotb
$ pip3 install cocotb-bus
$ pip3 install cocotb-test
$ pip3 install cocotbext-axi
$ pip3 install cocotbext-eth
$ pip3 install cocotbext-pcie
$ pip3 install pytest
$ pip3 install scapy
$ pip3 install tox
$ pip3 install pytest-xdist
$ pip3 install pytest-sugar

注意：安裝完package可能會(huì)報(bào)warning提示package不在PATH內(nèi)

此時(shí)需要將package添加到系統(tǒng)PATH中

$ sudo gedit ~/.bashrc

在bashrc中添加

export PAT

然后在終端source

$ sudo source ~/.bashrc

2.4 安裝雙系統(tǒng)Ubuntu 20.04
在win10下的機(jī)械盤上分了一塊200G大小的空間用于安裝新系統(tǒng)，事實(shí)證明200G有點(diǎn)小。

安裝過程見：

https://www.jianshu.com/p/54d9a3a695cc
https://zhuanlan.zhihu.com/p/135953477
https://www.zhihu.com/question/34611974

注意分區(qū)時(shí)：/boot在UEFI上是非必要的，系統(tǒng)直接安在efi上即可。

將分區(qū)/：的空間設(shè)置足夠大，這是總目錄，以后安裝的軟件默認(rèn)在此目錄下，vivado2019占用90多G，因此此塊空間或者/home的空間要足夠大。

安裝完系統(tǒng)配置環(huán)境，同時(shí)安裝vivado2019.2，裝在/opt/xilinx下即可。

三、Running test

單核的PC不建議（太慢了），建議上服務(wù)器跑。

確保Linux下所需要的package都安裝完畢。

在corundum根目錄下運(yùn)行仿真。

$ cd /path/to/corundum-master
$ tox
py39 installed: apipkg==1.5,attrs==21.2.0,cocotb==1.5.2,cocotb-bus==0.1.1,cocotb-test==0.2.0,cocotbext-axi==0.1.12,cocotbext-eth==0.1.14,cocotbext-pcie==0.1.10,execnet==1.8.0,iniconfig==1.1.1,packaging==20.9,pluggy==0.13.1,py==1.10.0,pyparsing==2.4.7,pytest==6.2.4,pytest-forked==1.3.0,pytest-split==0.1.5,pytest-xdist==2.2.1,scapy==2.4.5,toml==0.10.2
py39 run-test-pre: PYTHONHASHSEED='438068548'
py39 run-test: commands[0] | pytest -n auto
============================= test session starts ==============================
platform linux -- Python 3.9.2, pytest-6.2.4, py-1.10.0, pluggy-0.13.1
cachedir: .tox/py39/.pytest_cache
rootdir: /path/to/corundum, configfile: tox.ini, testpaths: fpga/common/tb, fpga/mqnic, fpga/mqnic_tdma
plugins: xdist-2.2.1, forked-1.3.0, cocotb-test-0.2.0, split-0.1.5
gw0 [38] / gw1 [38] / gw2 [38] / gw3 [38] / gw4 [38] / gw5 [38] / gw6 [38] / gw7 [38] / gw8 [38] / gw9 [38] / gw10 [38] / gw11 [38] / gw12 [38] / gw13 [38] / gw14 [38] / gw15 [38] / gw16 [38] / gw17 [38] / gw18 [38] / gw19 [38] / gw20 [38] / gw21 [38] / gw22 [38] / gw23 [38] / gw24 [38] / gw25 [38] / gw26 [38] / gw27 [38] / gw28 [38] / gw29 [38] / gw30 [38] / gw31 [38] / gw32 [38] / gw33 [38] / gw34 [38] / gw35 [38] / gw36 [38] / gw37 [38] / gw38 [38] / gw39 [38] / gw40 [38] / gw41 [38] / gw42 [38] / gw43 [38] / gw44 [38] / gw45 [38] / gw46 [38] / gw47 [38] / gw48 [38] / gw49 [38] / gw50 [38] / gw51 [38] / gw52 [38] / gw53 [38] / gw54 [38] / gw55 [38] / gw56 [38] / gw57 [38] / gw58 [38] / gw59 [38] / gw60 [38] gw0 [38] / gw1 [38] / gw2 [38] / gw3 [38] / gw4 [38] / gw5 [38] / gw6 [38] / gw7 [38] / gw8 [38] / gw9 [38] / gw10 [38] / gw11 [38] / gw12 [38] / gw13 [38] / gw14 [38] / gw15 [38] / gw16 [38] / gw17 [38] / gw18 [38] / gw19 [38] / gw20 [38] / gw21 [38] / gw22 [38] / gw23 [38] / gw24 [38] / gw25 [38] / gw26 [38] / gw27 [38] / gw28 [38] / gw29 [38] / gw30 [38] / gw31 [38] / gw32 [38] / gw33 [38] / gw34 [38] / gw35 [38] / gw36 [38] / gw37 [38] / gw38 [38] / gw39 [38] / gw40 [38] / gw41 [38] / gw42 [38] / gw43 [38] / gw44 [38] / gw45 [38] / gw46 [38] / gw47 [38] / gw48 [38] / gw49 [38] / gw50 [38] / gw51 [38] / gw52 [38] / gw53 [38] / gw54 [38] / gw55 [38] / gw56 [38] / gw57 [38] / gw58 [38] / gw59 [38] / gw60 [38] / gw61 [38] / gw62 [38] / gw63 [38]
...................................... [100%]
======================== 38 passed in 792.90s (0:13:12) ========================
___________________________________ summary ____________________________________
py39: commands succeeded
congratulations :)

我在虛擬機(jī)里面跑的大概時(shí)間37分鐘。。。

$ cd /path/to/corundum-master
$ pytest -n auto (多核)
$ pytest (單核)

$ cd /path/to/corundum/
$ pytest -n auto
Test session starts (platform: linux, Python 3.9.2, pytest 6.2.2, pytest-sugar 0.9.4)
rootdir: /path/to/corundum, configfile: tox.ini, testpaths: fpga/common/tb, fpga/mqnic, fpga/mqnic_tdma
plugins: parallel-0.1.0, cocotb-test-0.2.0, split-0.1.6.dev1+g97d96c2, sugar-0.9.4, xdist-2.2.1, forked-1.3.0, metadata-1.11.0, html-3.1.1, flake8-1.0.7, cov-2.11.1
gw0 [38] / gw1 [38] / gw2 [38] / gw3 [38] / gw4 [38] / gw5 [38] / gw6 [38] / gw7 [38] / gw8 [38] / gw9 [38] / gw10 [38] / gw11 [38] / gw12 [38] / gw13 [38] / gw14 [38] / gw15 [38] / gw16 [38] / gw17 [38] / gw18 [38] / gw19 [38] / gw20 [38] / gw21 [38] / gw22 [38] / gw23 [38] / gw24 [38] / gw25 [38] / gw26 [38] / gw27 [38] / gw28 [38] / gw29 [38] / gw30 [38] / gw31 [38] / gw32 [38] / gw33 [38] / gw34 [38] / gw35 [38] / gw36 [38] / gw37 [38] / gw38 [38] / gw39 [38] / gw40 [38] / gw41 [38] / gw42 [38] / gw43 [38] / gw44 [38] / gw45 [38] / gw46 [38] / gw47 [38] / gw48 [38] / gw49 [38] / gw50 [38] / gw51 [38] / gw52 [38] / gw53 [38] / gw54 [38] / gw55 [38] / gw56 [38] / gw57 [38] / gw58 [38] / gw59 [38] / gw60 [38] gw0 [38] / gw1 [38] / gw2 [38] / gw3 [38] / gw4 [38] / gw5 [38] / gw6 [38] / gw7 [38] / gw8 [38] / gw9 [38] / gw10 [38] / gw11 [38] / gw12 [38] / gw13 [38] / gw14 [38] / gw15 [38] / gw16 [38] / gw17 [38] / gw18 [38] / gw19 [38] / gw20 [38] / gw21 [38] / gw22 [38] / gw23 [38] / gw24 [38] / gw25 [38] / gw26 [38] / gw27 [38] / gw28 [38] / gw29 [38] / gw30 [38] / gw31 [38] / gw32 [38] / gw33 [38] / gw34 [38] / gw35 [38] / gw36 [38] / gw37 [38] / gw38 [38] / gw39 [38] / gw40 [38] / gw41 [38] / gw42 [38] / gw43 [38] / gw44 [38] / gw45 [38] / gw46 [38] / gw47 [38] / gw48 [38] / gw49 [38] / gw50 [38] / gw51 [38] / gw52 [38] / gw53 [38] / gw54 [38] / gw55 [38] / gw56 [38] / gw57 [38] / gw58 [38] / gw59 [38] / gw60 [38] / gw61 [38] / gw62 [38] / gw63 [38]

fpga/common/tb/tdma_scheduler/test_tdma_scheduler.py ? 3% ▍
fpga/common/tb/tdma_ber/test_tdma_ber.py ? 5% ▌
fpga/common/tb/queue_manager/test_queue_manager.py ? 8% ▊
fpga/common/tb/cpl_queue_manager/test_cpl_queue_manager.py ? 11% █▏
fpga/common/tb/rx_checksum/test_rx_checksum.py ?? 18% █▉
fpga/common/tb/rx_hash/test_rx_hash.py ?? 24% ██▍
fpga/common/tb/tdma_ber_ch/test_tdma_ber_ch.py ? 21% ██▏
fpga/common/tb/tx_checksum/test_tx_checksum.py ?? 29% ██▉
fpga/mqnic/VCU108/fpga_10g/tb/fpga_core/test_fpga_core.py ? 32% ███▎
fpga/mqnic/ExaNIC_X10/fpga/tb/fpga_core/test_fpga_core.py ? 34% ███▌
fpga/mqnic/AU50/fpga_100g/tb/fpga_core/test_fpga_core.py ? 37% ███▊

######## SNIP ########

fpga/mqnic/fb2CG/fpga_25g/tb/fpga_core/test_fpga_core.py ? 92% █████████▎
...ic/ADM_PCIE_9V3/fpga_25g/tb/fpga_core/test_fpga_core.py ? 95% █████████▌
...nic_tdma/VCU118/fpga_10g/tb/fpga_core/test_fpga_core.py ? 97% █████████▊
...ma/ADM_PCIE_9V3/fpga_10g/tb/fpga_core/test_fpga_core.py ? 100% ██████████

Results (797.62s):
38 passed

電腦還在跑。。。

審核編輯：郭婷