Performance¶
Environment¶
| Instance | Nodes | CPU | Memory | Storage | Network | Description |
| Master | 3 | 32 | 32 GB | 120 GB SSD | 10 Gb/s | |
| MetaNode | 10 | 32 | 32 GB | 16 x 1TB SSD | 10 Gb/s | hybrid deployment |
| DataNode | 10 | 32 | 32 GB | 16 x 1TB SSD | 10 Gb/s | hybrid deployment |
IO Performance and Scalability¶
IO Performance and benchmark scalability test by fio.
1. Sequential Read¶
Bandwidth
| 1 Process | 4 Processes | 16 Processes | 64 Processes | |
| 1 Client | 148.000 | 626.000 | 1129.000 | 1130.000 |
| 2 Clients | 284.000 | 1241.000 | 2258.000 | 2260.000 |
| 4 Clients | 619.000 | 2640.000 | 4517.000 | 4515.000 |
| 8 Clients | 1193.000 | 4994.000 | 9006.000 | 9034.000 |
IOPS
| 1 Process | 4 Processes | 16 Processes | 64 Processes | |
| 1 Client | 1180.000 | 5007.000 | 9031.000 | 9040.000 |
| 2 Clients | 2275.000 | 9924.000 | 18062.000 | 18081.000 |
| 4 Clients | 4954.000 | 21117.000 | 36129.000 | 36112.000 |
| 8 Clients | 9531.000 | 39954.000 | 72048.000 | 72264.000 |
Latency
| 1 Process | 4 Processes | 16 Processes | 64 Processes | |
| 1 Client | 842.200 | 794.340 | 1767.310 | 7074.550 |
| 2 Clients | 874.255 | 801.690 | 1767.370 | 7071.715 |
| 4 Clients | 812.363 | 760.702 | 1767.710 | 7077.065 |
| 8 Clients | 837.707 | 799.851 | 1772.620 | 7076.967 |
2. Sequential Write¶
Bandwidth
| 1 Process | 4 Processes | 16 Processes | 64 Processes | |
| 1 Client | 52.200 | 226.000 | 956.000 | 1126.000 |
| 2 Clients | 104.500 | 473.000 | 1763.000 | 2252.000 |
| 4 Clients | 225.300 | 1015.000 | 2652.000 | 3472.000 |
| 8 Clients | 480.600 | 1753.000 | 3235.000 | 3608.000 |
IOPS
| 1 Process | 4 Processes | 16 Processes | 64 Processes | |
| 1 Client | 417 | 1805 | 7651 | 9004 |
| 2 Clients | 835 | 3779 | 14103 | 18014 |
| 4 Clients | 1801 | 8127 | 21216 | 27777 |
| 8 Clients | 3841 | 14016 | 25890 | 28860 |
Latency
| 1 Process | 4 Processes | 16 Processes | 64 Processes | |
| 1 Client | 2385.400 | 2190.210 | 2052.360 | 7081.320 |
| 2 Clients | 2383.610 | 2081.850 | 2233.790 | 7079.450 |
| 4 Clients | 2216.305 | 1947.688 | 2946.017 | 8842.903 |
| 8 Clients | 2073.921 | 2256.120 | 4787.496 | 17002.425 |
3. Random Read¶
Bandwidth
| 1 Process | 4 Processes | 16 Processes | 64 Processes | |
| 1 Client | 6.412 | 39.100 | 216.000 | 534.000 |
| 2 Clients | 14.525 | 88.100 | 409.000 | 1002.000 |
| 4 Clients | 33.242 | 200.200 | 705.000 | 1693.000 |
| 8 Clients | 59.480 | 328.300 | 940.000 | 2369.000 |
IOPS
| 1 Process | 4 Processes | 16 Processes | 64 Processes | |
| 1 Client | 1641 | 10240 | 56524.800 | 140288 |
| 2 Clients | 3718 | 23142.4 | 107212.8 | 263168 |
| 4 Clients | 8508 | 52428.8 | 184627.2 | 443392 |
| 8 Clients | 15222 | 85072.8 | 246681.6 | 621056 |
Latency
| 1 Process | 4 Processes | 16 Processes | 64 Processes | |
| 1 Client | 603.580 | 395.420 | 287.510 | 466.320 |
| 2 Clients | 532.840 | 351.815 | 303.460 | 497.100 |
| 4 Clients | 469.025 | 317.140 | 355.105 | 588.847 |
| 8 Clients | 524.709 | 382.862 | 530.811 | 841.985 |
4. Random Write¶
Bandwidth
| 1 Process | 4 Processes | 16 Processes | 64 Processes | |
| 1 Client | 3.620 | 17.500 | 118.000 | 318.000 |
| 2 Clients | 7.540 | 44.800 | 230.000 | 476.000 |
| 4 Clients | 16.245 | 107.700 | 397.900 | 636.000 |
| 8 Clients | 39.274 | 208.100 | 487.100 | 787.100 |
IOPS
| 1 Process | 4 Processes | 16 Processes | 64 Processes | |
| 1 Client | 926.000 | 4476.000 | 31027.200 | 83251.200 |
| 2 Clients | 1929.000 | 11473.000 | 60313.600 | 124620.800 |
| 4 Clients | 4156.000 | 27800.000 | 104243.200 | 167014.400 |
| 8 Clients | 10050.000 | 53250.000 | 127692.800 | 206745.600 |
Latency
| 1 Process | 4 Processes | 16 Processes | 64 Processes | |
| 1 Client | 1073.150 | 887.570 | 523.820 | 784.030 |
| 2 Clients | 1030.010 | 691.530 | 539.525 | 1042.685 |
| 4 Clients | 955.972 | 575.183 | 618.445 | 1552.205 |
| 8 Clients | 789.883 | 598.393 | 1016.185 | 2506.424 |
Metadata Performance and Scalability¶
Metadata performance and scalability benchmark test by mdtest.
Dir Creation
| 1 Process | 4 Processes | 16 Processes | 64 Processes | |
| 1 Client | 448.618 | 2421.001 | 14597.97 | 43055.15 |
| 2 Clients | 956.947 | 5917.576 | 28930.431 | 72388.765 |
| 4 Clients | 2027.02 | 13213.403 | 54449.056 | 104771.356 |
| 8 Clients | 4643.755 | 27416.904 | 89641.301 | 119542.62 |
Dir Removal
| 1 Process | 4 Processes | 16 Processes | 64 Processes | |
| 1 Client | 399.779 | 2118.005 | 12351.635 | 34903.672 |
| 2 Clients | 833.353 | 5176.812 | 24471.674 | 50242.973 |
| 4 Clients | 1853.617 | 11462.927 | 46413.313 | 91128.059 |
| 8 Clients | 4441.435 | 24133.617 | 74401.336 | 115013.557 |
Dir Stat
| 1 Process | 4 Processes | 16 Processes | 64 Processes | |
| 1 Client | 283232.761 | 1215309.524 | 4231088.104 | 12579177.02 |
| 2 Clients | 572834.143 | 2169669.058 | 8362749.217 | 18120970.71 |
| 4 Clients | 1263474.549 | 3333746.786 | 10160929.29 | 31874265.88 |
| 8 Clients | 2258670.069 | 8715752.83 | 22524794.98 | 77533648.04 |
File Creation
| 1 Process | 4 Processes | 16 Processes | 64 Processes | |
| 1 Client | 448.888 | 2400.803 | 13638.072 | 27785.947 |
| 2 Clients | 925.68 | 5664.166 | 25889.163 | 50434.484 |
| 4 Clients | 2001.137 | 12986.968 | 50330.952 | 91387.825 |
| 8 Clients | 4479.831 | 25933.437 | 86667.966 | 112746.199 |
File Removal
| 1 Process | 4 Processes | 16 Processes | 64 Processes | |
| 1 Client | 605.143 | 3678.138 | 18631.342 | 47035.912 |
| 2 Clients | 1301.151 | 8365.667 | 34005.25 | 64860.041 |
| 4 Clients | 3032.683 | 14017.426 | 50938.926 | 80692.761 |
| 8 Clients | 7170.386 | 32056.959 | 68761.908 | 88357.563 |
Tree Creation
| 1 Process | 4 Processes | 16 Processes | 64 Processes | |
| 1 Client | 305.778 | 229.562 | 86.299 | 23.917 |
| 2 Clients | 161.31 | 211.119 | 76.301 | 24.439 |
| 4 Clients | 260.562 | 223.153 | 81.209 | 23.867 |
| 8 Clients | 350.038 | 220.744 | 81.621 | 17.144 |
Tree Removal
| 1 Process | 4 Processes | 16 Processes | 64 Processes | |
| 1 Client | 137.462 | 70.881 | 31.235 | 7.057 |
| 2 Clients | 217.026 | 113.36 | 23.971 | 7.128 |
| 4 Clients | 231.622 | 113.539 | 30.626 | 7.299 |
| 8 Clients | 185.156 | 83.923 | 20.607 | 5.515 |
Integrity¶
- Linux Test Project / fs
Workload¶
database backup
Java application logs
code git repo
database systems
MyRocks, MySQL Innodb, HBase,
Scalability¶
- volume scalability: tens to millions of cfs volumes
- metadata scalability: a big volume with billions of files/directories